From 9daa76760767f8338f8130ae27789488394ec6b7 Mon Sep 17 00:00:00 2001 From: KnifeLemon Date: Fri, 20 Nov 2026 21:02:05 +0900 Subject: [PATCH 1/6] feat(ebooks): write PDB, PML, TCR, RB and SNB without calibre The built-in e-book engine now writes the older formats it already read, from the document model the other writers use: - PDB as PalmDOC: 4 KB PalmDOC-compressed text records, the PalmDOC header, the title as database name, a bookmark record per heading - TCR: a 256-entry dictionary (every byte used plus frequent pairs merged on a sample), the text coded by longest match - PML: ASCII with \aNNN / \UXXXX, chapters and sub-headings, styles, alignment, list markers, links and anchors, footnotes, PNG pictures in the _img folder reserved through the output allocator - RB: the XHTML page in zlib chunks, the info page, PNG pictures - SNB: book, contents and chapter XML, bzip2 text blocks (own encoder), file table and block tail Text is Windows-1252 when it fits, else UTF-8; the RB reader now reads an info page in UTF-8 too. Each writer stays within its reader's limits, checks for cancellation and writes through a temporary file. The edges cost 19 like AZW3 / MOBI, so calibre ranked first still takes EPUB to these formats. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- docs/ENGINES.md | 30 +- src/Filee.Engines/Ebooks/CalibreConverter.cs | 8 +- src/Filee.Engines/Ebooks/EbookConverter.cs | 43 +- .../Ebooks/Legacy/LegacyWriting.cs | 84 ++++ src/Filee.Engines/Ebooks/Legacy/PdbWriter.cs | 84 ++++ src/Filee.Engines/Ebooks/Legacy/PmlWriter.cs | 388 +++++++++++++++ .../Ebooks/Legacy/RocketBookReader.cs | 5 +- .../Ebooks/Legacy/RocketBookWriter.cs | 153 ++++++ src/Filee.Engines/Ebooks/Legacy/SnbWriter.cs | 359 ++++++++++++++ src/Filee.Engines/Ebooks/Legacy/TcrWriter.cs | 169 +++++++ src/Filee.Engines/Ebooks/Mobi/PalmDatabase.cs | 40 +- .../LegacyEbookWriterTests.cs | 454 ++++++++++++++++++ 13 files changed, 1793 insertions(+), 26 deletions(-) create mode 100644 src/Filee.Engines/Ebooks/Legacy/LegacyWriting.cs create mode 100644 src/Filee.Engines/Ebooks/Legacy/PdbWriter.cs create mode 100644 src/Filee.Engines/Ebooks/Legacy/PmlWriter.cs create mode 100644 src/Filee.Engines/Ebooks/Legacy/RocketBookWriter.cs create mode 100644 src/Filee.Engines/Ebooks/Legacy/SnbWriter.cs create mode 100644 src/Filee.Engines/Ebooks/Legacy/TcrWriter.cs create mode 100644 tests/Filee.Engines.Tests/LegacyEbookWriterTests.cs diff --git a/README.md b/README.md index 0349efeb3..bbf1f1935 100644 --- a/README.md +++ b/README.md @@ -177,7 +177,7 @@ To undo it, remove the entries in Windows Security → App & browser control → | **Presentations**: PPTX, PPTM, POTX, POTM, PPSX, PPT, ODP, OTP, FODP
PowerPoint 95 PPT, Keynote² | PDF, PNG, JPG, ODP, OTP, FODP, DOCX, HWPX; PPT → PPTX | | **HWP**: HWP, HWPX | PDF, PNG, DOCX, HWPX ↔ HWP, TXT, Markdown, HTML, PPTX and ODP slides | | **Text**: TXT, LOG, script source (PY, PAC, BAT, COMMAND; read, never run), Arduino and Processing sketches (INO, PDE), Markdown, HTML;
MHTML (saved web pages) read
Jupyter, reStructuredText, Org, MediaWiki, DocBook, Textile, OPML
LaTeX, AsciiDoc, Typst, man³ | UTF-8 TXT, PDF, DOCX, HWPX, EPUB, PPTX and ODP slides (a slide per heading)
each other, PPTX and ODP slides³ | -| **E-books**: EPUB, KEPUB, MOBI, AZW3, AZW, AZW4, FB2, CBZ, CBR, CB7, HTMLZ, TXTZ, CHM, PDB, PML, TCR, RB, SNB, OEB
LIT, LRF⁴ | PDF, EPUB, KEPUB (Kobo), DOCX, TXT, HTML, HWPX
MOBI, AZW3 | +| **E-books**: EPUB, KEPUB, MOBI, AZW3, AZW, AZW4, FB2, CBZ, CBR, CB7, HTMLZ, TXTZ, CHM, PDB, PML, TCR, RB, SNB, OEB
LIT, LRF⁴ | PDF, EPUB, KEPUB (Kobo), DOCX, TXT, HTML, HWPX
MOBI, AZW3, PDB, PML, TCR, RB, SNB | | **Video**⁵: MP4, MOV, MKV, WEBM, AVI, WMV, FLV, MPEG, Video CD (DAT), TS, M2TS, 3GP, OGV, VOB, …;
Wisenet WAVE / Nx Witness exports (NOV) read | MP4 (H.264, HEVC, AV1), WEBM, MOV, MKV, AVI, GIF, APNG, WEBP, MP3, M4A, 720p, split into parts;
subtitle tracks kept (MKV, MP4, MOV, WebM);
subtitles (SRT, ASS, WebVTT) | | **Audio**⁵: MP3, M4A, AAC, WAV, FLAC, OGG, OPUS, WMA, AIFF, AMR, APE, WavPack, TTA, DSD (DSF, DFF), DTS, AC3, E-AC3, …;
tracker modules (XM, IT, S3M) read | MP3, M4A (AAC, ALAC), M4R (iPhone ringtone), WAV, FLAC, OGG, OPUS, AAC, WavPack, TTA, W64, AC3, E-AC3 | | **Subtitles**: SRT, ASS/SSA, WebVTT, SAMI (SMI), LRC, KRC, TTML, MicroDVD, SCC | each other as UTF-8 (Korean CP949 files read correctly) | diff --git a/docs/ENGINES.md b/docs/ENGINES.md index 3ed3dc706..66c224872 100644 --- a/docs/ENGINES.md +++ b/docs/ENGINES.md @@ -33,7 +33,7 @@ Filee chooses engines automatically. Settings → *Engines* shows their status, | E-mail | built in (MimeKit, MsgReader) | **EML, Outlook MSG → HTML** (headers + body, inline pictures), **TXT**, **ZIP** (the attachments); **MSG → EML**; **MHTML → HTML** (the saved page in one file, pictures inlined; on to PDF, DOCX, HWPX, TXT, Markdown, EPUB from there) | | Signed files | built in (System.Security.Cryptography.Pkcs) | **P7M → the signed file inside** (CMS / PKCS #7, CAdES: Italian e-invoices `*.xml.p7m`, signed PDFs `*.pdf.p7m`), then on from its format (P7M → PDF → JPG); the signature isn't checked | | Data | built in (System.Text.Json, YamlDotNet, Tomlyn, System.Xml) | **JSON, JSON Lines, YAML, TOML, XML, CSV ↔ each other** (CSV for tables); **vCard, iCalendar ↔ CSV**; **property lists (.plist, XML and binary) → each of them and TXT**; FreeMASTER oscilloscope TXT → CSV; spreadsheets, contacts and events reach JSON / YAML / TOML / XML through CSV | -| **E-books** | built in | **EPUB, MOBI/AZW/AZW3/PRC, FB2, HTMLZ, TXTZ, CHM, PDB, PML, TCR, RB, SNB, OEB → EPUB, AZW3, MOBI, HWPX (→ PDF), TXT, HTML, Markdown, FB2, HTMLZ, TXTZ**; HTML, Markdown, TXT, DOCX → EPUB/AZW3/MOBI/FB2/HTMLZ/TXTZ; HTML → TXT; comics CBZ/CBR/CB7/CBT/CBC → PDF, EPUB, CBZ; PDF → CBZ; AZW4 → PDF; **EPUB ↔ KEPUB** (Kobo) | +| **E-books** | built in | **EPUB, MOBI/AZW/AZW3/PRC, FB2, HTMLZ, TXTZ, CHM, PDB, PML, TCR, RB, SNB, OEB → EPUB, AZW3, MOBI, PDB, PML, TCR, RB, SNB, HWPX (→ PDF), TXT, HTML, Markdown, FB2, HTMLZ, TXTZ**; HTML, Markdown, TXT, DOCX → EPUB/AZW3/MOBI/PDB/PML/TCR/RB/SNB/FB2/HTMLZ/TXTZ; HTML → TXT; comics CBZ/CBR/CB7/CBT/CBC → PDF, EPUB, CBZ; PDF → CBZ; AZW4 → PDF; **EPUB ↔ KEPUB** (Kobo) | | **Drawings** | built in | **Visio VSDX, ODG, OTG, FODG → PDF, SVG, PNG** with the slide renderer (a page per drawing page; see [Drawings](#drawings-visio-odg)); **XPS, OpenXPS → PDF, SVG, PNG** with Filee's own XPS reader ([XPS reader](#xps-and-openxps-reader)); **Excel chart templates (CRTX) → PDF, SVG, PNG** with the [chart drawer](#charts); a binary `.vsd` and a drawing with what the readers don't draw yet (charts, OLE objects, 3D) keep LibreOffice's route | | **DjVu** | built in | **DjVu (.djvu, .djv; one page or bundled) → PDF** (the layers kept: background JPEG, 1-bit masks; the hidden text searchable) and **PNG, JPG, TIFF** pictures of the pages (#346; see [DjVu](#djvu)) | | CAD (ACadSharp + built-in renderer) | built in (library) | **DWG ↔ DXF**; DWG/DXF → **PDF** (vector), **SVG**, PNG, JPG, WEBP, TIFF, BMP, GIF, ICO, AVIF | @@ -51,7 +51,7 @@ Filee chooses engines automatically. Settings → *Engines* shows their status, | Tesseract | **optional download** (~50 MB, 160 MB on disk; each language 0.4–10 MB when first used) | **text recognition (OCR)**: scans and pictures → searchable PDF, and their text into DOCX, TXT, Markdown and HWPX ("Recognize text" in the preset, see below) | | Apple Vision | part of macOS (Mac only) | **text recognition (OCR) on a Mac**, the same step as Tesseract without a download; Tesseract reads the languages Vision doesn't (see "Apple Vision on macOS") | | FFmpeg | **optional download** (~100 MB, 272 MB on disk) | **all video and audio**: video ↔ video (H.264, HEVC, AV1), video → animated GIF / APNG / WebP, a still frame or every frame, audio and subtitle extraction, audio ↔ audio (also APE, WavPack, TTA, DSD, DTS, ALAC, …), GIF, APNG and photos → MP4/WEBM/MOV, Live / Motion Photos → their video, a clip of a video → an iPhone Live Photo or Android Motion Photo (experimental), images → HEIC, clips or audio files joined into one, pictures → a slideshow MP4, a video split into parts every so often or into equal parts (copied, not encoded), copies without metadata (see "Media tools") | -| Calibre | **optional download** (~216 MB, 660 MB on disk) | rare e-book formats: LIT, LRF → EPUB (CHM, PDB, PML, RB, SNB, TCR, OEB too, below the built-in readers); EPUB → LIT, LRF, PDB, PML, RB, SNB, TCR (MOBI, AZW3 too, below the built-in writer); e-books → PDF (ranked above the HWPX writer with the Filee renderer off, see below) | +| Calibre | **optional download** (~216 MB, 660 MB on disk) | rare e-book formats: LIT, LRF → EPUB (CHM, PDB, PML, RB, SNB, TCR, OEB too, below the built-in readers); EPUB → LIT, LRF (MOBI, AZW3, PDB, PML, RB, SNB, TCR too, below the built-in writers); e-books → PDF (ranked above the HWPX writer with the Filee renderer off, see below) | | WeasyPrint | **optional download** (Windows only for now; ~32 MB, 65 MB on disk) | **HTML, Markdown, EML, MSG, MHTML → PDF** laid out as a browser prints them (headings, tables, bold, Arabic and Indic shaping), instead of the built-in route; nothing changes without it (see below) | | Whisper | **optional download** (Windows only; ~157 MB with model, ~160 MB on disk) | **Speech-to-text transcription**: video and audio → SRT subtitles or plain TXT in ~100 languages using OpenAI Whisper (whisper.cpp); runs locally and offline; needs FFmpeg | | ExifTool | **optional download** (~11 MB on Windows, ~8 MB elsewhere; 30 MB on disk) | **the Metadata tool**: reads every metadata field of pictures, PDFs, video and sound, and writes changes into a copy in the file's own format without re-encoding it (JPG, PNG, TIFF, WEBP, HEIC, AVIF, JPEG XL, GIF, PSD, camera RAW, PDF, MP4, MOV, M4A, 3GP, …; not MP3, FLAC, WAV, MKV, AVI or BMP, which it shows read-only). Without it the Metadata tool shows nothing and offers it | @@ -3665,7 +3665,7 @@ writer, so every e-book also reaches HWPX, PDF and images. The document model is naturally ("page2" before "page10"). → PDF has one page per picture sized like the picture (JPEGs are embedded as they are), → EPUB is fixed-layout, → CBZ repacks. PDF → CBZ renders the pages with PDFium at the preset DPI. - **Older formats** (`Ebooks/Legacy`, `Ebooks/Chm`, written from the formats' public descriptions), so calibre is - needed only for LIT and LRF: + needed only for LIT and LRF (reading and writing): - **CHM** (Compiled HTML Help): the ITSF header, the ITSP directory and its PMGL listing chunks, files stored or in the LZX section. The LZX decoder is Filee's own (`Chm/LzxDecoder.cs`): verbatim, aligned offset and uncompressed blocks, pretree deltas, repeat offsets, 16-bit realignment after each 32 KB frame, the reset interval and window @@ -3692,19 +3692,35 @@ writer, so every e-book also reaches HWPX, PDF and images. The document model is outside the file, trees with too many codes, matches before the start of the data, records out of order) is `InvalidDataException`. `DocumentFuzzTests` fuzzes the CHM, PDB, TCR, RB and SNB readers and the LZX decoder on its own. Test inputs are written in code, CHM's LZX section too (`LzxEncoder`, a test-only compressor). + - **Writers** (cost 19 like AZW3 and MOBI: they win in the default engine order, calibre's EPUB → them wins once + calibre is ranked first). Text is Windows-1252 when every character fits, else UTF-8 (PDB, TCR, the RB info page). + **PDB** is PalmDOC: the plain text in 4 KB PalmDOC-compressed records, the 16-byte header, the title as the + database name and a bookmark record (name and byte offset) per heading of the top three levels. **TCR**: a + dictionary of every byte the text uses plus pairs merged by frequency on a sample of up to 512 KB, the text coded + by longest match. **PML**: ASCII with `\aNNN` / `\UXXXX` characters, chapters `\x` and `\X1`-`\X4`, bold / italic / + underline / strike-through / super- and subscript, centred and right-aligned paragraphs, list markers, links and + anchors, footnotes in `` blocks, page breaks, rules; pictures as PNG in the `_img` folder, reserved + through the output allocator like Markdown's picture folder. **RB**: the XhtmlWriter page (no style sheet, + non-ASCII as character references) in 4 KB zlib chunks, the info page (TITLE, AUTHOR, BODY) and the pictures as + PNG. **SNB**: book.snbf, toc.snbf and a chapter file per top-level heading (paragraphs and pictures: SNB has no + styles), the text stream in 32 KB bzip2 blocks (Filee's own encoder), the binary stream, the file table and the + block tail. Each writer stays within the limits its reader applies (64 MB of text, 65,535 Palm records, 10,000 RB + files, 100,000 SNB files, 1 GB), checks for cancellation and writes through a temporary file, so a failed or + cancelled conversion leaves nothing behind. `LegacyEbookWriterTests` reads every format back with Filee's readers + and checks the headers, records, dictionaries and blocks. - Books with DRM (Adobe, Apple, Kindle) are refused with a clear message; Filee does not remove DRM. KFX and Topaz books are recognised and refused too. ## Calibre calibre's `ebook-convert` (GPL-3.0) is an optional download for the formats Filee does not read or write itself: -LIT and LRF → EPUB, and EPUB → LIT, LRF, PDB, PML, RB, SNB and TCR. CHM, PDB, PML, RB, SNB, TCR and OEB are read, -and MOBI and AZW3 written, by the built-in engine (see [E-books](#e-books)); calibre keeps its edges for them, below +LIT and LRF → EPUB, and EPUB → LIT and LRF. CHM, PDB, PML, RB, SNB, TCR and OEB are read, and MOBI, AZW3, PDB, PML, +RB, SNB and TCR written, by the built-in engine (see [E-books](#e-books)); calibre keeps its edges for them, below the built-in ones. EPUB is the hub, so for example FB2 → LIT is FB2 → EPUB (built in) → LIT (calibre). Its edges cost 20, so built-in routes always win where they exist. -calibre's other outputs (LIT, LRF, PDB, PML, RB, SNB, TCR) are kept: they cost nothing while calibre isn't installed, -and nothing built in writes them. +calibre's other outputs (LIT, LRF) are kept: they cost nothing while calibre isn't installed, and nothing built in +writes them. calibre also makes PDFs of EPUB, MOBI, AZW3, AZW, PRC, FB2, HTMLZ, TXTZ and the formats only it reads (one `ebook-convert` run: A4 with `--paper-size a4`, page numbers with `--pdf-page-numbers`, calibre's margins and font diff --git a/src/Filee.Engines/Ebooks/CalibreConverter.cs b/src/Filee.Engines/Ebooks/CalibreConverter.cs index 0bc9e8f53..1e3074bbc 100644 --- a/src/Filee.Engines/Ebooks/CalibreConverter.cs +++ b/src/Filee.Engines/Ebooks/CalibreConverter.cs @@ -1,7 +1,7 @@ // E-book formats Filee does not read or write itself, with calibre's ebook-convert (GPL-3.0, a separate program the -// user downloads in Settings → Engines): LIT and LRF → EPUB, and EPUB → LIT, LRF, PDB, PML, RB, SNB and TCR. CHM, -// PDB, PML, RB, SNB, TCR and OEB are read, and MOBI and AZW3 written (MobiWriter), by the built-in engine too; -// calibre's edges for them stay as the fallback a user can rank first. EPUB is the hub: the built-in engine handles +// user downloads in Settings → Engines): LIT and LRF → EPUB, and EPUB → LIT and LRF. CHM, PDB, PML, RB, SNB, TCR and +// OEB are read, and MOBI, AZW3, PDB, PML, RB, SNB and TCR written (MobiWriter, Ebooks/Legacy), by the built-in engine +// too; calibre's edges for them stay as the fallback a user can rank first. EPUB is the hub: the built-in engine handles // every other step, and calibre's edges cost more so built-in routes always win where they exist. // // calibre also makes PDFs of e-books, much faster than the built-in route (HWPX writer → rhwp) for long books (#127). @@ -33,7 +33,7 @@ public sealed partial class CalibreConverter : IConverter /// Formats only calibre reads. internal static readonly string[] OnlyCalibreReads = ["lit", "lrf"]; - /// Formats calibre writes (MOBI and AZW3 are written by the built-in engine too). + /// Formats calibre writes (all but LIT and LRF are written by the built-in engine too). internal static readonly string[] Outputs = ["mobi", "azw3", "lit", "lrf", "pdb", "pml", "rb", "snb", "tcr"]; /// Books calibre makes PDFs of: the ones the built-in engines read too, and . diff --git a/src/Filee.Engines/Ebooks/EbookConverter.cs b/src/Filee.Engines/Ebooks/EbookConverter.cs index 09c1e0383..f10d2e297 100644 --- a/src/Filee.Engines/Ebooks/EbookConverter.cs +++ b/src/Filee.Engines/Ebooks/EbookConverter.cs @@ -1,10 +1,11 @@ // Built-in e-book conversions, no external program: EPUB, MOBI / AZW / AZW3 / PRC, FB2, HTMLZ and TXTZ, and the // older formats CHM, PDB (PalmDOC, zTXT), PML, TCR, RB, SNB and OEB (Ebooks/Legacy, Ebooks/Chm) are read into // the HWPX document model and written as EPUB, HWPX (→ PDF / images through rhwp and PDFium), TXT, HTML, Markdown, -// FB2, HTMLZ, TXTZ, AZW3 (KF8) or MOBI (MOBI 6 + KF8); HTML, Markdown, TXT and DOCX become e-books too. Comics -// (CBZ / CBR / CB7 / CBT / CBC) become PDF, fixed-layout EPUB or CBZ, PDF becomes CBZ, and AZW4 (Print Replica) gives -// back its PDF. EPUB ↔ KEPUB (Kobo) adds or removes Kobo's sentence markup and keeps the rest of the book as it is. -// Formats only calibre reads or writes (LIT, LRF, PDB / LIT / LRF output, ...) are in CalibreConverter. +// FB2, HTMLZ, TXTZ, AZW3 (KF8), MOBI (MOBI 6 + KF8) or the older PDB (PalmDOC), PML, TCR, RB and SNB (Ebooks/Legacy); +// HTML, Markdown, TXT and DOCX become e-books too. Comics (CBZ / CBR / CB7 / CBT / CBC) become PDF, fixed-layout EPUB +// or CBZ, PDF becomes CBZ, and AZW4 (Print Replica) gives back its PDF. EPUB ↔ KEPUB (Kobo) adds or removes Kobo's +// sentence markup and keeps the rest of the book as it is. Formats only calibre reads or writes (LIT, LRF) are in +// CalibreConverter. using System.Text; using Filee.Core.Conversion; @@ -21,19 +22,26 @@ public sealed class EbookConverter : IConverter /// E-book formats read into the document model. internal static readonly string[] BookReaders = ["epub", "kepub", "mobi", "azw3", "azw", "prc", "fb2", "htmlz", "txtz", "chm", "pdb", "pml", "tcr", "rb", "snb", "oeb"]; + /// + /// The older formats written from the document model (Ebooks/Legacy). They cost like + /// AZW3 and MOBI: calibre writes them too, and a user who ranks calibre first gets calibre's books. + /// + internal static readonly string[] LegacyWriters = ["pdb", "pml", "tcr", "rb", "snb"]; + /// Formats written from the document model. - private static readonly string[] BookWriters = ["epub", "hwpx", "txt", "html", "md", "fb2", "htmlz", "txtz", "azw3", "mobi"]; + private static readonly string[] BookWriters = ["epub", "hwpx", "txt", "html", "md", "fb2", "htmlz", "txtz", "azw3", "mobi", .. LegacyWriters]; /// - /// Cost of → AZW3 / MOBI. In the default engine order it costs 19 + 9 against calibre's EPUB → AZW3 / MOBI at - /// 20 + 9, so the built-in writer wins; a user who ranks calibre first (20 + 0) gets calibre's books instead. + /// Cost of → AZW3 / MOBI and the . In the default engine order it costs 19 + 9 against + /// calibre's EPUB → AZW3 / MOBI / PDB / ... at 20 + 9, so the built-in writer wins; a user who ranks calibre first + /// (20 + 0) gets calibre's books instead. /// internal const int KindleCost = 19; /// Documents that become e-books (their other conversions belong to the document engines). private static readonly string[] DocumentReaders = ["html", "md", "txt", "docx"]; - private static readonly string[] EbookWriters = ["epub", "fb2", "htmlz", "txtz", "azw3", "mobi"]; + private static readonly string[] EbookWriters = ["epub", "fb2", "htmlz", "txtz", "azw3", "mobi", .. LegacyWriters]; private static readonly string[] Comics = ["cbz", "cbr", "cb7", "cbt", "cbc"]; @@ -45,11 +53,11 @@ public sealed class EbookConverter : IConverter [ // KEPUB → EPUB is below: the book is copied without Kobo's markup instead of being written anew. .. BookReaders.SelectMany(from => BookWriters.Where(to => to != from && !(from == "kepub" && to == "epub")) - .Select(to => new ConversionEdge(from, to, to == "md" ? 12 : to is "azw3" or "mobi" ? KindleCost : 10))), + .Select(to => new ConversionEdge(from, to, to == "md" ? 12 : to is "azw3" or "mobi" || LegacyWriters.Contains(to) ? KindleCost : 10))), // Every other format reaches KEPUB through EPUB. new("epub", "kepub"), new("kepub", "epub"), - .. DocumentReaders.SelectMany(from => EbookWriters.Select(to => new ConversionEdge(from, to, to is "azw3" or "mobi" ? KindleCost : 10))), + .. DocumentReaders.SelectMany(from => EbookWriters.Select(to => new ConversionEdge(from, to, to is "azw3" or "mobi" || LegacyWriters.Contains(to) ? KindleCost : 10))), // HTML → TXT has no other built-in route; HTML → Markdown is a fallback for when Pandoc is not installed. new("html", "txt"), new("html", "md", 14), @@ -201,6 +209,21 @@ private static void Write(Book book, ConversionStep step, string output, string case "mobi": MobiWriter.Write(book, output, step.InputPath, work, mobi6: true); break; + case "pdb": + PdbWriter.Write(book, output, step.InputPath, cancellationToken); + break; + case "pml": + PmlWriter.Write(book, output, step.Preset.Document.LeaveOutPictures ? null : new PmlPictures(output, step.Output, work), cancellationToken); + break; + case "tcr": + TcrWriter.Write(book, output, cancellationToken); + break; + case "rb": + RocketBookWriter.Write(book, output, step.InputPath, work, cancellationToken); + break; + case "snb": + SnbWriter.Write(book, output, step.InputPath, work, cancellationToken); + break; default: throw new NotSupportedException($"{step.From} → {step.To}"); } diff --git a/src/Filee.Engines/Ebooks/Legacy/LegacyWriting.cs b/src/Filee.Engines/Ebooks/Legacy/LegacyWriting.cs new file mode 100644 index 000000000..0fdd366f2 --- /dev/null +++ b/src/Filee.Engines/Ebooks/Legacy/LegacyWriting.cs @@ -0,0 +1,84 @@ +// Shared by the writers of the older e-book formats (PDB, TCR, PML, RB, SNB): the text encoding their readers +// expect, the size limit a written book stays within (the same one the readers apply, so a book Filee writes is a +// book Filee reads), and pictures as PNG for the formats whose devices show nothing else. + +using System.Text; +using Filee.Engines.Documents; +using Filee.Engines.Hwp.Hwpx; + +namespace Filee.Engines.Ebooks; + +internal static class LegacyWriting +{ + /// Windows-1252 that throws on characters it lacks (instead of writing '?'). + public static Encoding StrictWindows1252 + { + get + { + Encoding.RegisterProvider(CodePagesEncodingProvider.Instance); + return Encoding.GetEncoding(1252, EncoderFallback.ExceptionFallback, DecoderFallback.ExceptionFallback); + } + } + + /// Windows-1252 that writes '?' for characters it lacks (names in fixed-size fields). + public static Encoding Windows1252 + { + get + { + Encoding.RegisterProvider(CodePagesEncodingProvider.Instance); + return Encoding.GetEncoding(1252); + } + } + + /// + /// Text as Windows-1252 when every character has a byte there (what Palm and Psion readers were made for), else + /// as UTF-8 without a byte order mark (Filee's readers and calibre take valid UTF-8 first). + /// + public static (byte[] Bytes, Encoding Encoding) Encode(string text) + { + var windows1252 = StrictWindows1252; + try + { + return (windows1252.GetBytes(text), windows1252); + } + catch (EncoderFallbackException) + { + var utf8 = new UTF8Encoding(false); + return (utf8.GetBytes(text), utf8); + } + } + + /// A document as plain text () with line ends. + public static string PlainText(HDocument document, string newLine) => + PlainTextWriter.Write(document).Replace("\r\n", "\n").Replace("\n", newLine); + + /// Refuses a book whose text passes the limit the readers apply (64 MB): no reader would open it. + public static void CheckText(long bytes, string format) + { + var limit = DocumentLimits.Default.MaxPartBytes; + if (bytes > limit) + throw DocumentLimits.Exceeded($"The {format} book's text would be larger than {DocumentLimits.Megabytes(limit)}."); + } + + /// + /// The picture at as a PNG file (converted into when it is + /// another format), or null when it cannot be read. + /// + public static string? Png(string path, string folder) + { + if (EbookFiles.ImageMediaType(path) == "image/png") + return File.Exists(path) ? path : null; + try + { + using var image = new ImageMagick.MagickImage(path); + Directory.CreateDirectory(folder); + var file = Path.Combine(folder, $"{Guid.NewGuid():N}.png"); + image.Write(file, ImageMagick.MagickFormat.Png); + return file; + } + catch (Exception ex) when (ex is ImageMagick.MagickException or IOException or UnauthorizedAccessException) + { + return null; + } + } +} diff --git a/src/Filee.Engines/Ebooks/Legacy/PdbWriter.cs b/src/Filee.Engines/Ebooks/Legacy/PdbWriter.cs new file mode 100644 index 000000000..346ec0da1 --- /dev/null +++ b/src/Filee.Engines/Ebooks/Legacy/PdbWriter.cs @@ -0,0 +1,84 @@ +// Book → PDB as PalmDOC (TEXtREAd), from the PalmDOC format description: the book's text (PlainTextWriter, newline +// line ends, Windows-1252 or UTF-8 as LegacyWriting.Encode picks) in records of 4,096 bytes, each PalmDOC-compressed +// (the LZ77 variant MobiWriter uses too). Record 0 is the 16-byte PalmDOC header: compression 2, the text length, +// the number of text records, the record size and the reading position 0. After the text come bookmark records, +// one per heading of the top three levels, 16 bytes of name and the heading's byte offset in the text: Palm readers +// show them as the book's contents. The database name is the title. + +using System.Buffers.Binary; +using Filee.Engines.Hwp.Hwpx; + +namespace Filee.Engines.Ebooks; + +internal static class PdbWriter +{ + private const int RecordSize = 4096; + private const int BookmarkNameLength = 16; + + public static void Write(Book book, string outputPath, string sourcePath, CancellationToken cancellationToken = default) + { + var text = LegacyWriting.PlainText(book.Document, "\n"); + var (bytes, encoding) = LegacyWriting.Encode(text); + LegacyWriting.CheckText(bytes.Length, "PDB"); + + var records = new List { Header(bytes.Length, (bytes.Length + RecordSize - 1) / RecordSize) }; + for (var offset = 0; offset < bytes.Length; offset += RecordSize) + { + if ((offset & 0xFFFFF) == 0) + cancellationToken.ThrowIfCancellationRequested(); + records.Add(PalmDocCompression.Compress(bytes.AsSpan(offset, Math.Min(RecordSize, bytes.Length - offset)))); + } + foreach (var (name, position) in Bookmarks(book.Document, text, encoding)) + { + if (records.Count >= PalmDatabase.MaxRecords) + break; + records.Add(Bookmark(name, position)); + } + var file = PalmDatabase.Build(book.DisplayTitle(sourcePath), "TEXtREAd", records); + cancellationToken.ThrowIfCancellationRequested(); + WriterTempFile.Write(outputPath, stream => stream.Write(file)); + } + + /// Record 0: compression 2 (PalmDOC), unused, text length, text records, record size, position. + private static byte[] Header(int textLength, int textRecords) + { + var header = new byte[16]; + BinaryPrimitives.WriteUInt16BigEndian(header, 2); + BinaryPrimitives.WriteUInt32BigEndian(header.AsSpan(4), (uint)textLength); + BinaryPrimitives.WriteUInt16BigEndian(header.AsSpan(8), (ushort)textRecords); + BinaryPrimitives.WriteUInt16BigEndian(header.AsSpan(10), RecordSize); + return header; + } + + /// A bookmark record: the name in 16 bytes (Windows-1252, '?' for what it lacks, zero-padded) and the offset. + private static byte[] Bookmark(string name, long position) + { + var record = new byte[BookmarkNameLength + 4]; + var bytes = LegacyWriting.Windows1252.GetBytes(name); + bytes.AsSpan(0, Math.Min(bytes.Length, BookmarkNameLength - 1)).CopyTo(record); + BinaryPrimitives.WriteUInt32BigEndian(record.AsSpan(BookmarkNameLength), (uint)position); + return record; + } + + /// + /// The headings of the top three levels with the byte offset of their text in , found in + /// order (a heading the text doesn't hold as written, e.g. one with a line break, is left out). + /// + internal static IEnumerable<(string Name, long Position)> Bookmarks(HDocument document, string text, System.Text.Encoding encoding) + { + var from = 0; + long bytes = 0; + foreach (var heading in document.Sections.SelectMany(s => s.Blocks).OfType().Where(p => p.HeadingLevel is >= 1 and <= 3)) + { + var title = HDocumentWalker.Text(heading.Inlines).Trim(); + if (title.Length == 0) + continue; + var at = text.IndexOf(title, from, StringComparison.Ordinal); + if (at < 0) + continue; + bytes += encoding.GetByteCount(text.AsSpan(from, at - from)); + from = at; + yield return (title, bytes); + } + } +} diff --git a/src/Filee.Engines/Ebooks/Legacy/PmlWriter.cs b/src/Filee.Engines/Ebooks/Legacy/PmlWriter.cs new file mode 100644 index 000000000..6de8154d0 --- /dev/null +++ b/src/Filee.Engines/Ebooks/Legacy/PmlWriter.cs @@ -0,0 +1,388 @@ +// Book → PML (Palm Markup Language, the source format of eReader books), from the published PML reference that +// PmlReader follows too. The file is plain ASCII with CR LF line ends: characters of Windows-1252 above 127 become +// \aNNN, other characters \UXXXX (outside the Basic Multilingual Plane, which \U can't reach, U+FFFD), a backslash \\ +// and '<' \a060 (so text never looks like a block). +// * Each paragraph is a line, with an empty line after it. Level-1 headings are chapters (\x … \x), levels 2 to 5 +// \X1 … \X4 (deeper ones \X4): eReader lists both as the book's contents. +// * Bold \B, italic \i, underline \u, strike-through \o, superscript \Sp and subscript \Sb are opened and closed as +// the runs change, and all closed at the end of every line (PML codes left open run into the next paragraph). +// * Centred and right-aligned paragraphs are \c … \c and \r … \r; list items get their marker and \T="n%" indents; +// page breaks are \p; a paragraph that is only a horizontal line is a rule (\w="50%"). +// * Links inside the book are \q="#id" … \q to anchors \Q="id" (links to the web keep their text only); notes are +// \Fn="fnN"N\Fn with the note in a block at the end of the file. +// * Pictures are \m="img0001.png": PNG files (eReader shows no other format) in the "_img" folder beside the +// file, where eReader and PmlReader look for them, reserved through the output allocator like any output. +// * Table rows are lines of their cells separated by " | ". + +using System.Globalization; +using System.Text; +using Filee.Core.Conversion; +using Filee.Engines.Documents; +using Filee.Engines.Hwp.Hwpx; + +namespace Filee.Engines.Ebooks; + +internal sealed class PmlWriter +{ + private const string NewLine = "\r\n"; + + /// Windows-1252 bytes 128-255 by character, for \aNNN. + private static readonly Dictionary UpperWindows1252 = BuildUpperWindows1252(); + + private readonly StringBuilder _pml = new(); + private readonly Func? _picture; + private readonly List> _notes = []; + private readonly ListNumbers _numbers = new(); + private readonly List _open = []; + private readonly long _maxChars = DocumentLimits.Default.MaxPartBytes; + private readonly CancellationToken _cancellationToken; + private bool _inLink; + + private PmlWriter(Func? picture, CancellationToken cancellationToken) + { + _picture = picture; + _cancellationToken = cancellationToken; + } + + /// Where pictures go, or null to leave them out. + public static void Write(Book book, string outputPath, PmlPictures? pictures, CancellationToken cancellationToken = default) + { + var bytes = Encoding.ASCII.GetBytes(ToPml(book.Document, pictures is null ? null : pictures.Add, cancellationToken)); + WriterTempFile.Write(outputPath, stream => stream.Write(bytes)); + } + + /// The PML text of a document; gives the file name of a picture, or null. + internal static string ToPml(HDocument document, Func? picture, CancellationToken cancellationToken = default) + { + Nesting.Limit(document); + var writer = new PmlWriter(picture, cancellationToken); + foreach (var section in document.Sections) + writer.Blocks(section.Blocks); + // Notes may hold notes of their own: they are numbered on as they come. + for (var i = 0; i < writer._notes.Count; i++) + { + writer._pml.Append($"").Append(NewLine); + writer.Blocks(writer._notes[i]); + writer._pml.Append("").Append(NewLine).Append(NewLine); + writer.CheckSize(); + } + return writer._pml.ToString(); + } + + private void Blocks(List blocks) + { + foreach (var block in blocks) + { + _cancellationToken.ThrowIfCancellationRequested(); + switch (block) + { + case HParagraph paragraph: + Paragraph(paragraph); + break; + case HTable table: + Table(table); + break; + } + CheckSize(); + } + } + + private void Paragraph(HParagraph paragraph) + { + var inlines = HDocumentWalker.InlinesOf(paragraph.Inlines).ToList(); + var visible = inlines.Any(i => i is HImage or HNote or HTextBox || i is HText { Text: var t } && t.Trim().Length > 0); + if (paragraph.PageBreakBefore && paragraph.HeadingLevel != 1 && _pml.Length > 0) + _pml.Append("\\p").Append(NewLine); + if (!visible) + { + if (inlines.Any(i => i is HShape { Kind: HShapeKind.Line })) + _pml.Append("\\w=\"50%\"").Append(NewLine).Append(NewLine); + return; + } + if (paragraph.HeadingLevel > 0) + { + // \x starts a new page in eReader by itself. + var code = paragraph.HeadingLevel == 1 ? "\\x" : $"\\X{Math.Min(paragraph.HeadingLevel, 5) - 1}"; + _pml.Append(code); + Inlines(paragraph.Inlines, heading: true); + CloseAll(); + _pml.Append(code).Append(NewLine).Append(NewLine); + return; + } + var align = paragraph.Format.Align switch + { + HAlign.Center => "\\c", + HAlign.Right => "\\r", + _ => null, + }; + _pml.Append(align); + if (paragraph.List is { } list) + { + _pml.Append($"\\T=\"{Math.Min(5 * (list.Level + 1), 50)}%\""); + Text((list.Numbered ? _numbers.Next(list) : "•") + " "); + } + Inlines(paragraph.Inlines, heading: false); + CloseAll(); + _pml.Append(align).Append(NewLine).Append(NewLine); + } + + /// A row per line, its cells separated by " | "; header rows bold. + private void Table(HTable table) + { + foreach (var row in table.Rows) + { + if (row.Header) + Open("B"); + for (var c = 0; c < row.Cells.Count; c++) + { + if (c > 0) + Text(" | "); + CellInlines(row.Cells[c].Blocks, row.Header); + } + CloseAll(); + _pml.Append(NewLine).Append(NewLine); + } + } + + /// A cell's paragraphs (and those of tables inside it) on one line, separated by a space. + private void CellInlines(List blocks, bool bold) + { + var first = true; + foreach (var block in blocks) + { + if (block is HTable inner) + { + foreach (var cell in inner.Rows.SelectMany(r => r.Cells)) + CellInlines(cell.Blocks, bold); + continue; + } + if (block is not HParagraph paragraph) + continue; + if (!first) + Text(" "); + first = false; + Inlines(paragraph.Inlines.Where(i => i is not HImage).ToList(), heading: true, bold); + } + } + + /// On one line: line breaks become spaces (a heading or a table cell). + /// Bold whatever the runs say (table header cells). + private void Inlines(List inlines, bool heading, bool bold = false) + { + foreach (var inline in inlines) + { + switch (inline) + { + case HText text: + Format(bold ? text.Format with { Bold = true } : text.Format); + Text(text.Text); + break; + case HTab: + Text(" "); + break; + case HLineBreak: + if (heading) + { + Text(" "); + } + else + { + CloseAll(); + _pml.Append(NewLine); + } + break; + case HLink link: + Link(link, heading, bold); + break; + case HBookmark bookmark: + _pml.Append($"\\Q=\"{Id(bookmark.Name)}\""); + break; + case HImage image: + if (_picture?.Invoke(image.Path) is { } name) + _pml.Append($"\\m=\"{name}\""); + break; + case HNote note: + _notes.Add(note.Blocks); + var number = _notes.Count.ToString(CultureInfo.InvariantCulture); + if (_inLink) + Text($"[{number}]"); // \Fn is a link of its own and would end the one it is in + else + _pml.Append($"\\Fn=\"fn{number}\"").Append(number).Append("\\Fn"); + break; + case HTextBox box: + foreach (var paragraph in box.Blocks.OfType()) + { + Text(" "); + Inlines(paragraph.Inlines, heading: true, bold); + } + break; + } + } + } + + /// A link inside the book is \q="#id" … \q; one to anywhere else keeps its text. + private void Link(HLink link, bool heading, bool bold) + { + if (_inLink || !link.Target.StartsWith('#') || link.Target.Length < 2) + { + Inlines(link.Content, heading, bold); + return; + } + _pml.Append($"\\q=\"#{Id(link.Target[1..])}\""); + _inLink = true; + Inlines(link.Content, heading: true, bold); + _inLink = false; + _pml.Append("\\q"); + } + + /// Opens and closes the style codes so that exactly those of are open. + private void Format(HCharFormat format) + { + var wanted = new List(5); + if (format.Bold is true) + wanted.Add("B"); + if (format.Italic is true) + wanted.Add("i"); + if (format.Underline is true) + wanted.Add("u"); + if (format.Strike is true) + wanted.Add("o"); + if (format.Superscript is true) + wanted.Add("Sp"); + else if (format.Subscript is true) + wanted.Add("Sb"); + for (var k = _open.Count - 1; k >= 0; k--) + { + if (!wanted.Contains(_open[k])) + { + _pml.Append('\\').Append(_open[k]); + _open.RemoveAt(k); + } + } + foreach (var code in wanted) + Open(code); + } + + private void Open(string code) + { + if (_open.Contains(code)) + return; + _pml.Append('\\').Append(code); + _open.Add(code); + } + + private void CloseAll() + { + for (var k = _open.Count - 1; k >= 0; k--) + _pml.Append('\\').Append(_open[k]); + _open.Clear(); + } + + /// Text with PML's escapes; control characters are left out. + private void Text(string text) + { + for (var i = 0; i < text.Length; i++) + { + var c = text[i]; + if (c == '\\') + _pml.Append("\\\\"); + else if (c == '<') + _pml.Append("\\a060"); + else if (c is >= ' ' and <= '~') + _pml.Append(c); + else if (c is '\t' or '\n' or '\r') + _pml.Append(' '); + else if (c < ' ' || c == '\u007F') + continue; + else if (c == '\u00AD') + _pml.Append("\\-"); + else if (UpperWindows1252.TryGetValue(c, out var code)) + _pml.Append("\\a").Append(code.ToString("000", CultureInfo.InvariantCulture)); + else if (char.IsSurrogate(c)) + { + if (char.IsHighSurrogate(c) && i + 1 < text.Length && char.IsLowSurrogate(text[i + 1])) + i++; + _pml.Append("\\UFFFD"); + } + else + _pml.Append("\\U").Append(((int)c).ToString("X4", CultureInfo.InvariantCulture)); + } + } + + /// An anchor name PML can quote: letters, digits, '-', '_' and '.'. + private static string Id(string name) + { + var id = new string(name.Select(c => char.IsAsciiLetterOrDigit(c) || c is '-' or '_' or '.' ? c : '_').ToArray()); + return id.Length > 0 ? id : "_"; + } + + /// Refuses a book whose PML passes the limit PmlReader applies to the file. + private void CheckSize() + { + if (_pml.Length > _maxChars) + throw DocumentLimits.Exceeded($"The PML book would be larger than {DocumentLimits.Megabytes(_maxChars)}."); + } + + private static Dictionary BuildUpperWindows1252() + { + var map = new Dictionary(); + var encoding = LegacyWriting.StrictWindows1252; + for (var b = 0x80; b <= 0xFF; b++) + { + try + { + map.TryAdd(encoding.GetString([(byte)b])[0], b); + } + catch (DecoderFallbackException) + { + // Bytes Windows-1252 leaves undefined. + } + } + map.Remove('\u00AD'); // the soft hyphen is \- of its own + return map; + } +} + +/// +/// The pictures of a PML book: PNG files in the "<name>_img" folder beside it, created on the first picture +/// and reserved through the allocator ("book_img", "book_img (2)" or skipped, as the preset's conflict rule says). An +/// intermediate PML file (a step of a longer route) keeps the folder right beside itself, so the next step finds it. +/// +internal sealed class PmlPictures(string pml, IOutputAllocator output, string workDirectory) +{ + public const string FolderSuffix = "_img"; + + private readonly string _convertFolder = Path.Combine(workDirectory, "pml-pictures-" + Guid.NewGuid().ToString("N")[..8]); + private readonly Dictionary _names = new(StringComparer.OrdinalIgnoreCase); + private string? _folder; + private bool _skipped; + + /// The file name of the picture at in the folder, or null when it can't be read. + public string? Add(string path) + { + if (_names.TryGetValue(path, out var known)) + return known; + if (LegacyWriting.Png(path, _convertFolder) is not { } file || Folder() is not { } folder) + return null; + var name = $"img{_names.Count + 1:0000}.png"; + File.Copy(file, Path.Combine(folder, name), overwrite: true); + return _names[path] = name; + } + + private string? Folder() + { + if (_folder is not null || _skipped) + return _folder; + var beside = Path.GetDirectoryName(Path.GetFullPath(pml))!; + var folder = output.Allocate("", FolderSuffix); + if (folder is not null && !string.Equals(Path.GetDirectoryName(Path.GetFullPath(folder)), beside, StringComparison.OrdinalIgnoreCase)) + folder = Path.Combine(beside, Path.GetFileNameWithoutExtension(pml) + FolderSuffix); + if (folder is null) + { + _skipped = true; + return null; + } + Directory.CreateDirectory(folder); + return _folder = folder; + } +} diff --git a/src/Filee.Engines/Ebooks/Legacy/RocketBookReader.cs b/src/Filee.Engines/Ebooks/Legacy/RocketBookReader.cs index bd2e0a016..1ea5e7526 100644 --- a/src/Filee.Engines/Ebooks/Legacy/RocketBookReader.cs +++ b/src/Filee.Engines/Ebooks/Legacy/RocketBookReader.cs @@ -2,7 +2,8 @@ // the little-endian offset of its table of contents is at 0x18. The table lists the files: a 32-byte name, length, // offset and flags (1 compressed, 2 encrypted, 8 the info page). A compressed file is its chunk count, its size, // the size of each chunk and the chunks, each a zlib stream of up to 4 KB of the file. The info page is text lines -// of KEY=value (TITLE, AUTHOR, BODY = the HTML page to read); the pages are read through HtmlReader. +// of KEY=value (TITLE, AUTHOR, BODY = the HTML page to read), UTF-8 when valid, else Windows-1252; the pages are read +// through HtmlReader. // Encrypted books are refused as DRM. using System.Buffers.Binary; @@ -61,7 +62,7 @@ public static Book ReadBook(string path, string workFolder, CancellationToken ca throw DocumentLimits.Exceeded($"The RB file unpacks to more than {DocumentLimits.Megabytes(limits.MaxPackageBytes)}."); if ((flags & 8) != 0) { - foreach (var line in Encoding.Latin1.GetString(bytes).Split('\n', '\r')) + foreach (var line in MobiReader.DecodeText(bytes).Split('\n', '\r')) if (line.IndexOf('=') is var equals and > 0) info.TryAdd(line[..equals].Trim(), line[(equals + 1)..].Trim()); continue; diff --git a/src/Filee.Engines/Ebooks/Legacy/RocketBookWriter.cs b/src/Filee.Engines/Ebooks/Legacy/RocketBookWriter.cs new file mode 100644 index 000000000..5da0cd1dc --- /dev/null +++ b/src/Filee.Engines/Ebooks/Legacy/RocketBookWriter.cs @@ -0,0 +1,153 @@ +// Book → RB (NuvoMedia Rocket eBook), the layout RocketBookReader reads: a 0x128-byte header (B0 0C B0 0C, version +// 2, "NUVO", the offset of the table of contents at 0x18 and the file's length at 0x1C, little-endian), the table +// (the file count, then per file a 32-byte name, its length, offset and flags) and the files: +// * the info page (flag 8, stored): TITLE, AUTHOR and BODY lines, in Windows-1252 when they fit, else UTF-8; +// * the book as one HTML page (flag 1, compressed): the chunk count, the page's size, each chunk's size, then the +// chunks, each a zlib stream of 4 KB of the page. The page is the XhtmlWriter body (headings, bold, italic, +// links, anchors, lists, tables, notes) without a style sheet, every character beyond ASCII written as a +// character reference, so it reads the same in any character set; +// * the pictures (stored) as PNG, "img0001.png" and on. + +using System.Buffers.Binary; +using System.IO.Compression; +using System.Text; +using Filee.Engines.Documents; +using Filee.Engines.Hwp.Hwpx; + +namespace Filee.Engines.Ebooks; + +internal static class RocketBookWriter +{ + private const int TocOffset = 0x128; + private const int ChunkSize = 4096; + private const int NameLength = 32; + + /// Files RocketBookReader takes in one book. + private const int MaxFiles = 10_000; + + private const string PageName = "book.html"; + + public static void Write(Book book, string outputPath, string sourcePath, string workFolder, CancellationToken cancellationToken = default) + { + var limits = DocumentLimits.Default; + var title = book.DisplayTitle(sourcePath); + var convert = EbookFiles.NewFolder(workFolder, "rb-pictures"); + var pictures = new List<(string Name, string File)>(); + var names = new Dictionary(StringComparer.OrdinalIgnoreCase); + string? Picture(string path) + { + if (names.TryGetValue(path, out var known)) + return known; + if (pictures.Count >= MaxFiles - 2 || LegacyWriting.Png(path, convert) is not { } file) + return null; + var name = $"img{pictures.Count + 1:0000}.png"; + pictures.Add((name, file)); + return names[path] = name; + } + + var body = XhtmlWriter.Write(book.Document, new XhtmlOptions { ImageSource = Picture }).Single().Body; + cancellationToken.ThrowIfCancellationRequested(); + var html = Ascii($"\n\n{XhtmlWriter.Escape(title)}\n\n\n{body}\n\n"); + LegacyWriting.CheckText(html.Length, "RB"); + + var info = new StringBuilder(); + info.Append("TITLE=").Append(OneLine(title)).Append("\r\n"); + if (book.Metadata.Authors.Count > 0) + info.Append("AUTHOR=").Append(OneLine(string.Join(", ", book.Metadata.Authors))).Append("\r\n"); + info.Append("BODY=").Append(PageName).Append("\r\n"); + + var files = new List<(string Name, byte[] Data, int Flags)> + { + (Path.ChangeExtension(PageName, ".info"), LegacyWriting.Encode(info.ToString()).Bytes, 8), + (PageName, Compress(Encoding.ASCII.GetBytes(html), cancellationToken), 1), + }; + long total = files.Sum(f => (long)f.Data.Length); + foreach (var (name, file) in pictures) + { + cancellationToken.ThrowIfCancellationRequested(); + var data = File.ReadAllBytes(file); + if ((total += data.Length) > limits.MaxPackageBytes) + throw DocumentLimits.Exceeded($"The RB book would be larger than {DocumentLimits.Megabytes(limits.MaxPackageBytes)}."); + files.Add((name, data, 0)); + } + var rb = Container(files); + WriterTempFile.Write(outputPath, stream => stream.Write(rb)); + } + + /// The header, the table of contents and the files, in order. + internal static byte[] Container(IReadOnlyList<(string Name, byte[] Data, int Flags)> files) + { + var dataOffset = TocOffset + 4 + files.Count * 44; + var length = dataOffset + files.Sum(f => (long)f.Data.Length); + if (length > int.MaxValue) + throw DocumentLimits.Exceeded("The RB book would be larger than 2 GB."); + var rb = new byte[length]; + ((ReadOnlySpan)[0xB0, 0x0C, 0xB0, 0x0C, 0x02, 0x00]).CopyTo(rb); + "NUVO"u8.CopyTo(rb.AsSpan(6)); + BinaryPrimitives.WriteUInt32LittleEndian(rb.AsSpan(0x18), TocOffset); + BinaryPrimitives.WriteUInt32LittleEndian(rb.AsSpan(0x1C), (uint)length); + BinaryPrimitives.WriteUInt32LittleEndian(rb.AsSpan(TocOffset), (uint)files.Count); + var offset = dataOffset; + for (var i = 0; i < files.Count; i++) + { + var (name, data, flags) = files[i]; + var entry = rb.AsSpan(TocOffset + 4 + i * 44, 44); + var nameBytes = Encoding.ASCII.GetBytes(name); + nameBytes.AsSpan(0, Math.Min(nameBytes.Length, NameLength - 1)).CopyTo(entry); + BinaryPrimitives.WriteUInt32LittleEndian(entry[32..], (uint)data.Length); + BinaryPrimitives.WriteUInt32LittleEndian(entry[36..], (uint)offset); + BinaryPrimitives.WriteUInt32LittleEndian(entry[40..], (uint)flags); + data.CopyTo(rb, offset); + offset += data.Length; + } + return rb; + } + + /// A compressed file: chunk count, size, chunk sizes, then the zlib chunks of 4 KB each. + internal static byte[] Compress(byte[] data, CancellationToken cancellationToken = default) + { + var chunks = new List(); + for (var offset = 0; offset < data.Length; offset += ChunkSize) + { + if ((offset & 0xFFFFF) == 0) + cancellationToken.ThrowIfCancellationRequested(); + using var chunk = new MemoryStream(); + using (var zlib = new ZLibStream(chunk, CompressionLevel.Optimal, leaveOpen: true)) + zlib.Write(data, offset, Math.Min(ChunkSize, data.Length - offset)); + chunks.Add(chunk.ToArray()); + } + using var output = new MemoryStream(); + var number = new byte[4]; + void Le32(int value) + { + BinaryPrimitives.WriteInt32LittleEndian(number, value); + output.Write(number); + } + Le32(chunks.Count); + Le32(data.Length); + foreach (var chunk in chunks) + Le32(chunk.Length); + foreach (var chunk in chunks) + output.Write(chunk); + return output.ToArray(); + } + + /// The page with every character beyond ASCII as a numeric character reference. + private static string Ascii(string html) + { + var sb = new StringBuilder(html.Length + html.Length / 8); + for (var i = 0; i < html.Length; i++) + { + var c = html[i]; + if (c < 0x80) + sb.Append(c); + else if (char.IsHighSurrogate(c) && i + 1 < html.Length && char.IsLowSurrogate(html[i + 1])) + sb.Append("&#").Append(char.ConvertToUtf32(c, html[++i])).Append(';'); + else if (!char.IsSurrogate(c)) + sb.Append("&#").Append((int)c).Append(';'); + } + return sb.ToString(); + } + + private static string OneLine(string text) => text.Replace('\r', ' ').Replace('\n', ' ').Trim(); +} diff --git a/src/Filee.Engines/Ebooks/Legacy/SnbWriter.cs b/src/Filee.Engines/Ebooks/Legacy/SnbWriter.cs new file mode 100644 index 000000000..0af4de59d --- /dev/null +++ b/src/Filee.Engines/Ebooks/Legacy/SnbWriter.cs @@ -0,0 +1,359 @@ +// Book → SNB (Shanda Bambook), the container SnbReader reads (all numbers big-endian): +// * the 44-byte header: "SNBP000B", the version words, the file count, the file table's size unpacked and packed, +// the sizes of the binary and the text stream, a reserved word; +// * the file table (zlib): per file its attributes (0x41000000 text, 0x01000000 binary), the offset of its name and +// its size, then the names, each ended by a zero byte, sorted by name; +// * the binary stream (the pictures, stored), then the text stream (the XML files) in 32 KB blocks, each +// bzip2-compressed with Filee's own encoder (a full block that bzip2 doesn't make smaller is stored); +// * the tail (zlib): the offset of every block (binary blocks first), then per file its block and its offset in it; +// the last 16 bytes are the tail's size, its offset and "SNBP000B". +// The files: snbf/book.snbf (title, authors, language, publisher, description, cover), snbf/toc.snbf (the +// chapters in order) and one snbc/chNNNN.snbc per chapter: its title and paragraphs and pictures +// (JPEG or PNG in snbc/images). SNB text has no styles: chapters start at the book's top heading level, other +// headings, list items (with their markers), table rows (cells separated by " | ") and notes (numbered, at the end of +// their chapter) are paragraphs. + +using System.Buffers.Binary; +using System.Globalization; +using System.IO.Compression; +using System.Text; +using System.Xml; +using System.Xml.Linq; +using Filee.Core.Presets; +using Filee.Engines.Archives; +using Filee.Engines.Documents; +using Filee.Engines.Hwp.Hwpx; + +namespace Filee.Engines.Ebooks; + +internal static class SnbWriter +{ + private const int BlockSize = 0x8000; + private const int TextAttributes = 0x41000000; + private const int BinaryAttributes = 0x01000000; + private static readonly byte[] Magic = "SNBP000B"u8.ToArray(); + + /// A file of the container. + internal sealed record SnbFile(string Name, byte[] Data, bool Text); + + public static void Write(Book book, string outputPath, string sourcePath, string workFolder, CancellationToken cancellationToken = default) + { + Nesting.Limit(book.Document); + var title = book.DisplayTitle(sourcePath); + var convert = EbookFiles.NewFolder(workFolder, "snb-pictures"); + var files = new List(); + var pictures = new Dictionary(StringComparer.OrdinalIgnoreCase); + string? Picture(string path) + { + if (pictures.TryGetValue(path, out var known)) + return known; + if (EbookFiles.CommonImage(path, convert, gifAndSvg: false) is not ({ } file, { } mediaType)) + return null; + var name = $"img{pictures.Count + 1:0000}.{(mediaType == "image/png" ? "png" : "jpg")}"; + files.Add(new SnbFile("snbc/images/" + name, File.ReadAllBytes(file), Text: false)); + return pictures[path] = name; + } + + var cover = book.Metadata.CoverImage is { } coverFile && File.Exists(coverFile) ? Picture(coverFile) : null; + var chapters = Chapters(book.Document, title, Picture, cancellationToken); + var toc = new XElement("body"); + for (var i = 0; i < chapters.Count; i++) + { + var name = $"ch{i + 1:0000}.snbc"; + toc.Add(new XElement("chapter", new XAttribute("src", name), Clean(chapters[i].Title))); + files.Add(new SnbFile("snbc/" + name, Xml(new XElement("snbc", + new XElement("head", new XElement("title", Clean(chapters[i].Title))), + new XElement("body", chapters[i].Content))), Text: true)); + } + files.Add(new SnbFile("snbf/toc.snbf", Xml(new XElement("toc-snbf", + new XElement("head", new XElement("chapters", chapters.Count.ToString(CultureInfo.InvariantCulture))), toc)), Text: true)); + files.Add(new SnbFile("snbf/book.snbf", Xml(new XElement("book-snbf", new XAttribute("version", "1.0"), + new XElement("head", + new XElement("name", Clean(title)), + new XElement("author", Clean(string.Join(", ", book.Metadata.Authors))), + new XElement("language", book.DisplayLanguage().ToUpperInvariant()), + new XElement("rights"), + new XElement("publisher", Clean(book.Metadata.Publisher ?? "")), + new XElement("generator", "Filee"), + new XElement("created", DateTime.Now.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture)), + new XElement("abstract", Clean(book.Metadata.Description ?? "")), + new XElement("cover", cover ?? "")))), Text: true)); + + var snb = Container(files, cancellationToken); + WriterTempFile.Write(outputPath, stream => stream.Write(snb)); + } + + // ───────────────────────── Chapters ───────────────────────── + + /// The content before the first heading (left out when there is none). + private sealed record Chapter(string Title, List Content, bool Leading = false); + + /// + /// The chapters: a new one at every heading of the book's top level (1 when there is one); content before the + /// first is a chapter of its own, named after the book. + /// + private static List Chapters(HDocument document, string bookTitle, Func picture, CancellationToken cancellationToken) + { + var blocks = document.Sections.SelectMany(s => s.Blocks).ToList(); + var levels = blocks.OfType().Where(p => p.HeadingLevel > 0 && HDocumentWalker.Text(p.Inlines).Length > 0).Select(p => p.HeadingLevel).ToList(); + var top = levels.Count > 0 ? levels.Min() : 0; + var chapters = new List(); + var current = new Chapter(bookTitle, [], Leading: true); + var notes = new List(); + var numbers = new ListNumbers(); + + void Close() + { + for (var i = 0; i < notes.Count; i++) + current.Content.Add(TextElement($"[{i + 1}] {notes[i]}")); + notes.Clear(); + if (current.Content.Count > 0 || !current.Leading) + chapters.Add(current); + } + + foreach (var block in blocks) + { + cancellationToken.ThrowIfCancellationRequested(); + if (block is HParagraph { HeadingLevel: > 0 } heading && heading.HeadingLevel == top && HDocumentWalker.Text(heading.Inlines) is { Length: > 0 } headingText) + { + Close(); + current = new Chapter(headingText, []); + continue; + } + switch (block) + { + case HParagraph paragraph: + { + var marker = paragraph.List is { } list ? new string(' ', list.Level * 2) + (list.Numbered ? numbers.Next(list) : "•") + " " : ""; + Paragraph(paragraph.Inlines, marker, current.Content, notes, picture); + break; + } + case HTable table: + foreach (var row in table.Rows) + { + var cells = row.Cells.Select(c => string.Join(" ", c.Blocks.OfType().Select(p => HDocumentWalker.Text(p.Inlines)).Where(t => t.Length > 0))); + if (string.Join(" | ", cells) is var line && line.Replace("|", "").Trim().Length > 0) + current.Content.Add(TextElement(line)); + } + break; + } + } + Close(); + if (chapters.Count == 0) + chapters.Add(new Chapter(bookTitle, [])); + return chapters; + } + + /// A paragraph's lines as <text> elements, its pictures as <img> between them. + private static void Paragraph(List inlines, string marker, List content, List notes, Func picture) + { + var line = new StringBuilder(marker); + void Flush() + { + if (line.ToString().Trim().Length > 0) + content.Add(TextElement(line.ToString().TrimEnd())); + line.Clear(); + } + void Walk(List list) + { + foreach (var inline in list) + { + switch (inline) + { + case HText text: + line.Append(text.Text); + break; + case HTab: + line.Append('\t'); + break; + case HLineBreak: + Flush(); + break; + case HLink link: + Walk(link.Content); + break; + case HImage image: + if (picture(image.Path) is { } name) + { + Flush(); + content.Add(new XElement("img", name)); + } + break; + case HNote note: + notes.Add(string.Join(" ", note.Blocks.OfType().Select(p => HDocumentWalker.Text(p.Inlines)).Where(t => t.Length > 0))); + line.Append('[').Append(notes.Count).Append(']'); + break; + case HTextBox box: + Flush(); + foreach (var paragraph in box.Blocks.OfType()) + { + Walk(paragraph.Inlines); + Flush(); + } + break; + } + } + } + Walk(inlines); + Flush(); + } + + /// A <text> paragraph, its text in CDATA as Bambook books have it. + private static XElement TextElement(string text) + { + var clean = Clean(text); + return new XElement("text", clean.Contains("]]>", StringComparison.Ordinal) ? new XText(clean) : new XCData(clean)); + } + + /// Text without the characters XML 1.0 forbids and without line breaks. + private static string Clean(string text) + { + var sb = new StringBuilder(text.Length); + for (var i = 0; i < text.Length; i++) + { + var c = text[i]; + if (char.IsHighSurrogate(c) && i + 1 < text.Length && char.IsLowSurrogate(text[i + 1])) + sb.Append(c).Append(text[++i]); + else if (c is '\r' or '\n') + sb.Append(' '); + else if (XmlConvert.IsXmlChar(c)) + sb.Append(c); + } + return sb.ToString(); + } + + private static byte[] Xml(XElement root) + { + using var stream = new MemoryStream(); + using (var xml = XmlWriter.Create(stream, new XmlWriterSettings { Encoding = new UTF8Encoding(false) })) + new XDocument(new XDeclaration("1.0", "utf-8", null), root).Save(xml); + return stream.ToArray(); + } + + // ───────────────────────── Container ───────────────────────── + + /// The SNB container of (sorted by name here). + internal static byte[] Container(List files, CancellationToken cancellationToken = default) + { + var limits = DocumentLimits.Default; + files = [.. files.OrderBy(f => f.Name, StringComparer.Ordinal)]; + if (files.Count > limits.MaxPackageEntries) + throw DocumentLimits.Exceeded($"The SNB book would have more than {DocumentLimits.Count(limits.MaxPackageEntries)} files."); + if (files.Sum(f => (long)f.Data.Length) > limits.MaxPackageBytes) + throw DocumentLimits.Exceeded($"The SNB book would be larger than {DocumentLimits.Megabytes(limits.MaxPackageBytes)}."); + + // Where each file lies in its stream, and the file table. + using var binary = new MemoryStream(); + using var text = new MemoryStream(); + using var table = new MemoryStream(); + using var names = new MemoryStream(); + var placed = new List<(bool Text, long Offset)>(); + foreach (var file in files) + { + var stream = file.Text ? text : binary; + placed.Add((file.Text, stream.Length)); + stream.Write(file.Data); + Be32(table, file.Text ? TextAttributes : BinaryAttributes); + Be32(table, (int)names.Length); + Be32(table, file.Data.Length); + names.Write(Encoding.UTF8.GetBytes(file.Name)); + names.WriteByte(0); + } + names.WriteTo(table); + if (table.Length > limits.MaxPartBytes) + throw DocumentLimits.Exceeded($"The SNB book's file table would be larger than {DocumentLimits.Megabytes(limits.MaxPartBytes)}."); + var packedTable = Zlib(table.ToArray()); + + var blocks = TextBlocks(text, cancellationToken); + var textSize = (int)text.Length; // padded by TextBlocks when it had to be + var binaryBlocks = (int)((binary.Length + BlockSize - 1) / BlockSize); + + using var output = new MemoryStream(); + output.Write(Magic); + Be32(output, 0x00008000); + Be32(output, 0x00A3A3A3); + Be32(output, 0); + Be32(output, files.Count); + Be32(output, (int)table.Length); + Be32(output, packedTable.Length); + Be32(output, (int)binary.Length); + Be32(output, textSize); + Be32(output, 0); + output.Write(packedTable); + + using var tail = new MemoryStream(); + var binaryStart = output.Length; + for (var i = 0; i < binaryBlocks; i++) + Be32(tail, (int)(binaryStart + (long)i * BlockSize)); + binary.WriteTo(output); + foreach (var block in blocks) + { + Be32(tail, (int)output.Length); + output.Write(block); + if (output.Length > int.MaxValue - 1024) + throw DocumentLimits.Exceeded("The SNB book would be larger than 2 GB."); + } + foreach (var (isText, offset) in placed) + { + Be32(tail, (int)(offset / BlockSize) + (isText ? binaryBlocks : 0)); + Be32(tail, (int)(offset % BlockSize)); + } + var packedTail = Zlib(tail.ToArray()); + var tailOffset = (int)output.Length; + output.Write(packedTail); + Be32(output, packedTail.Length); + Be32(output, tailOffset); + output.Write(Magic); + return output.ToArray(); + } + + /// + /// The text stream in 32 KB blocks, each bzip2-compressed, or stored when that doesn't make a full block smaller + /// (a block shorter than 32 KB is read as compressed, so a short last block that bzip2 can't shrink is padded + /// with spaces to a full stored one: the files end before the padding). + /// + private static List TextBlocks(MemoryStream text, CancellationToken cancellationToken) + { + var blocks = new List(); + for (long offset = 0; offset < text.Length; offset += BlockSize) + { + cancellationToken.ThrowIfCancellationRequested(); + var length = (int)Math.Min(BlockSize, text.Length - offset); + var block = text.GetBuffer().AsSpan((int)offset, length); + using var packed = new MemoryStream(); + using (var bzip2 = new Bzip2Writer(packed, ArchiveLevel.Normal, cancellationToken, leaveOpen: true, threads: 1)) + { + bzip2.Write(block); + bzip2.Finish(); + } + if (packed.Length < BlockSize) + { + blocks.Add(packed.ToArray()); + continue; + } + if (length < BlockSize) + { + text.Position = text.Length; + text.Write(Enumerable.Repeat((byte)' ', BlockSize - length).ToArray()); + block = text.GetBuffer().AsSpan((int)offset, BlockSize); + } + blocks.Add(block.ToArray()); + } + return blocks; + } + + private static void Be32(Stream stream, int value) + { + Span bytes = stackalloc byte[4]; + BinaryPrimitives.WriteInt32BigEndian(bytes, value); + stream.Write(bytes); + } + + private static byte[] Zlib(byte[] data) + { + using var output = new MemoryStream(); + using (var zlib = new ZLibStream(output, CompressionLevel.Optimal, leaveOpen: true)) + zlib.Write(data); + return output.ToArray(); + } +} diff --git a/src/Filee.Engines/Ebooks/Legacy/TcrWriter.cs b/src/Filee.Engines/Ebooks/Legacy/TcrWriter.cs new file mode 100644 index 000000000..d9d216b5e --- /dev/null +++ b/src/Filee.Engines/Ebooks/Legacy/TcrWriter.cs @@ -0,0 +1,169 @@ +// Book → TCR (the text compression of Psion e-book readers), from the format description TcrReader follows: the +// "!!8-Bit!!" signature, 256 dictionary entries (a length byte and up to 255 bytes each), then one byte per entry +// for the whole text. The text is the book's plain text (PlainTextWriter, CR LF line ends) in Windows-1252, or UTF-8 +// when it holds characters Windows-1252 lacks. +// +// The dictionary: every byte value the text uses gets an entry of its own (so any text can be written), and the +// codes left over are filled by pair merging on a sample of the text (up to 512 KB in slices across the book): the +// most frequent pair of neighbouring entries becomes a new entry, until all 256 codes are taken or no pair repeats. +// The text is then written with the longest entry that matches at each position. + +using Filee.Engines.Hwp.Hwpx; + +namespace Filee.Engines.Ebooks; + +internal static class TcrWriter +{ + private static readonly byte[] Signature = "!!8-Bit!!"u8.ToArray(); + + /// Bytes of the text the dictionary is learnt from, in slices across it. + private const int SampleBytes = 512 * 1024; + + private const int SampleSlices = 16; + private const int MaxEntryLength = 255; + + public static void Write(Book book, string outputPath, CancellationToken cancellationToken = default) + { + var (bytes, _) = LegacyWriting.Encode(LegacyWriting.PlainText(book.Document, "\r\n")); + LegacyWriting.CheckText(bytes.Length, "TCR"); + var file = Compress(bytes, cancellationToken); + WriterTempFile.Write(outputPath, stream => stream.Write(file)); + } + + /// The TCR file of : signature, dictionary, codes. + internal static byte[] Compress(ReadOnlySpan text, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + var entries = Dictionary(text, cancellationToken); + using var output = new MemoryStream(text.Length / 2 + 1024); + output.Write(Signature); + for (var i = 0; i < 256; i++) + { + var entry = i < entries.Count ? entries[i] : []; + output.WriteByte((byte)entry.Length); + output.Write(entry); + } + + // The entries starting with each byte, longest first: the first that matches is the longest match. + var byFirst = new List[256]; + for (var code = 0; code < entries.Count; code++) + (byFirst[entries[code][0]] ??= []).Add(code); + foreach (var list in byFirst) + list?.Sort((a, b) => entries[b].Length.CompareTo(entries[a].Length)); + + var codes = new byte[Math.Min(text.Length, 1 << 16)]; + var count = 0; + for (var i = 0; i < text.Length;) + { + if (count == codes.Length) + { + cancellationToken.ThrowIfCancellationRequested(); + output.Write(codes, 0, count); + count = 0; + } + foreach (var code in byFirst[text[i]]!) + { + var entry = entries[code]; + if (text[i..].StartsWith(entry)) + { + codes[count++] = (byte)code; + i += entry.Length; + break; + } + } + } + output.Write(codes, 0, count); + return output.ToArray(); + } + + /// + /// The dictionary: an entry for every byte value of , then entries made by merging the most + /// frequent neighbouring pair of a sample, as long as codes are left and a pair occurs at least three times. + /// + internal static List Dictionary(ReadOnlySpan text, CancellationToken cancellationToken) + { + var present = new bool[256]; + foreach (var b in text) + present[b] = true; + // Symbols 0-255 are bytes, 256 and up merged entries; -1 separates the slices of the sample. + var symbols = new List(); + var entries = new List(); + for (var b = 0; b < 256; b++) + { + symbols.Add([(byte)b]); + if (present[b]) + entries.Add([(byte)b]); + } + if (entries.Count >= 256 || text.Length < 4) + return entries; + + var sample = Sample(text); + var length = sample.Length; + var counts = new int[512 * 512]; + while (entries.Count < 256 && symbols.Count < 512) + { + cancellationToken.ThrowIfCancellationRequested(); + Array.Clear(counts); + int best = -1, bestCount = 2; + for (var i = 0; i + 1 < length; i++) + { + var (a, b) = (sample[i], sample[i + 1]); + if (a < 0 || b < 0) + continue; + var key = a * 512 + b; + var n = ++counts[key]; + // A run "aaa" counts twice but merges once; good enough to rank the pairs. + if (n > bestCount && symbols[a].Length + symbols[b].Length <= MaxEntryLength) + (best, bestCount) = (key, n); + } + if (best < 0) + break; + var (left, right) = (best / 512, best % 512); + var merged = new byte[symbols[left].Length + symbols[right].Length]; + symbols[left].CopyTo(merged, 0); + symbols[right].CopyTo(merged, symbols[left].Length); + var symbol = symbols.Count; + symbols.Add(merged); + entries.Add(merged); + + var write = 0; + for (var read = 0; read < length; read++) + { + if (read + 1 < length && sample[read] == left && sample[read + 1] == right) + { + sample[write++] = symbol; + read++; + } + else + { + sample[write++] = sample[read]; + } + } + length = write; + } + return entries; + } + + /// Up to of the text as symbols, in slices from across it separated by -1. + private static int[] Sample(ReadOnlySpan text) + { + if (text.Length <= SampleBytes) + { + var whole = new int[text.Length]; + for (var i = 0; i < text.Length; i++) + whole[i] = text[i]; + return whole; + } + var slice = SampleBytes / SampleSlices; + var sample = new int[SampleSlices * (slice + 1)]; + var at = 0; + for (var s = 0; s < SampleSlices; s++) + { + var start = (int)((long)(text.Length - slice) * s / (SampleSlices - 1)); + foreach (var b in text.Slice(start, slice)) + sample[at++] = b; + sample[at++] = -1; + } + return sample; + } +} diff --git a/src/Filee.Engines/Ebooks/Mobi/PalmDatabase.cs b/src/Filee.Engines/Ebooks/Mobi/PalmDatabase.cs index 91b5b3370..e0984f172 100644 --- a/src/Filee.Engines/Ebooks/Mobi/PalmDatabase.cs +++ b/src/Filee.Engines/Ebooks/Mobi/PalmDatabase.cs @@ -1,6 +1,6 @@ // The Palm database (PDB) container of MOBI, AZW, AZW3, AZW4 and PRC books: a 78-byte header with the type and -// creator ("BOOKMOBI", "TEXtREAd"), then a table of record offsets. Written from the PalmOS database format -// description; all numbers are big-endian. +// creator ("BOOKMOBI", "TEXtREAd"), then a table of record offsets; read here, and written by Build for PalmDOC books +// (PdbWriter). Written from the PalmOS database format description; all numbers are big-endian. using System.Buffers.Binary; using System.Text; @@ -61,6 +61,42 @@ public ReadOnlySpan Record(int index) public byte[] RecordArray(int index) => Record(index).ToArray(); + /// Most records a database holds: the record count is a 16-bit number. + public const int MaxRecords = ushort.MaxValue; + + /// + /// A database file of : the header with the name (at most 31 bytes, Latin-1, others as + /// '_'), the creation and modification dates (seconds since 1904, as PalmOS counts), the type and creator, then + /// the record list (offset, no attributes, unique id) and the records after two bytes of padding. + /// + public static byte[] Build(string name, string typeCreator, IReadOnlyList records) + { + if (records.Count is 0 or > MaxRecords) + throw Documents.DocumentLimits.Exceeded($"A Palm database holds at most {Documents.DocumentLimits.Count(MaxRecords)} records."); + var headerLength = 78 + records.Count * 8 + 2; + var length = headerLength + records.Sum(r => (long)r.Length); + if (length > int.MaxValue) + throw Documents.DocumentLimits.Exceeded("The Palm database would be larger than 2 GB."); + var file = new byte[length]; + var bytes = name.Select(c => c is >= ' ' and <= '~' or >= ' ' and <= 'ÿ' ? (byte)c : (byte)'_').Take(31).ToArray(); + (bytes.Length > 0 ? bytes : "Book"u8.ToArray()).CopyTo(file, 0); + var now = (uint)(DateTimeOffset.UtcNow - new DateTimeOffset(1904, 1, 1, 0, 0, 0, TimeSpan.Zero)).TotalSeconds; + BinaryPrimitives.WriteUInt32BigEndian(file.AsSpan(36), now); + BinaryPrimitives.WriteUInt32BigEndian(file.AsSpan(40), now); + Encoding.ASCII.GetBytes(typeCreator).AsSpan(0, 8).CopyTo(file.AsSpan(60)); + BinaryPrimitives.WriteUInt32BigEndian(file.AsSpan(68), (uint)records.Count); // the next unique id + BinaryPrimitives.WriteUInt16BigEndian(file.AsSpan(76), (ushort)records.Count); + var offset = headerLength; + for (var i = 0; i < records.Count; i++) + { + BinaryPrimitives.WriteUInt32BigEndian(file.AsSpan(78 + i * 8), (uint)offset); + BinaryPrimitives.WriteUInt32BigEndian(file.AsSpan(78 + i * 8 + 4), (uint)i & 0x00FFFFFF); + records[i].CopyTo(file, offset); + offset += records[i].Length; + } + return file; + } + // Big-endian helpers with bounds checks: damaged books must fail with a clear error, not an exception deep inside. public static int U16(ReadOnlySpan data, int offset) => diff --git a/tests/Filee.Engines.Tests/LegacyEbookWriterTests.cs b/tests/Filee.Engines.Tests/LegacyEbookWriterTests.cs new file mode 100644 index 000000000..1cb86ca37 --- /dev/null +++ b/tests/Filee.Engines.Tests/LegacyEbookWriterTests.cs @@ -0,0 +1,454 @@ +// The older e-book formats written by the built-in engine (PDB as PalmDOC, PML, TCR, RB, SNB): each book is built in +// code, written, and read back with Filee's own reader (text, headings, styles, links, pictures and metadata where +// the format holds them), the files' structure is checked against the format descriptions (headers, record and file +// counts, dictionaries, blocks), and the routes take the built-in writers ahead of calibre unless calibre is ranked +// first. Limits refuse with error.document_too_complex, and a cancelled write leaves no file behind. + +using System.Buffers.Binary; +using System.Text; +using Filee.Core.Conversion; +using Filee.Core.Presets; +using Filee.Engines.Documents; +using Filee.Engines.Ebooks; +using Filee.Engines.Hwp.Hwpx; +using ImageMagick; + +namespace Filee.Engines.Tests; + +public class LegacyEbookWriterTests(EngineFixture fx) : IClassFixture +{ + private static readonly string[] OptionalEngines = ["pandoc", "libreoffice", "calibre"]; + private const string Title = "The Lighthouse Keeper"; + private const string Author = "Ada Lovelace"; + private const string Line1 = "Ships passed safely through the narrow strait."; + private const string Accented = "Café, naïve — “quoted” €5"; + private const string Korean = "등대지기의 밤"; + + private static CancellationToken Token => TestContext.Current.CancellationToken; + + /// + /// A book of two chapters with a sub-heading, bold and italic text, a picture, a link to the second chapter, a + /// list, a table, a footnote and characters from beyond ASCII (Korean too unless ). + /// + private Book SampleBook(bool latinOnly = false) + { + var folder = fx.NewFolder(); + using (var jpeg = new MagickImage(MagickColors.Teal, 40, 30)) + jpeg.Write(Path.Combine(folder, "pic.jpg"), MagickFormat.Jpeg); + var html = $""" + {Title} +

Chapter One

+

The lamp was lit at dusk every evening. {Accented}{(latinOnly ? "" : " " + Korean)}

+

+

Go to the second chapter. Back\slash and <footnote> text.

+
  • First item
  • Second item
+
NameValue
Star42
+

A Section

+

Water is H2O.

+

Chapter Two

+

{Line1}

+ + """; + var source = Path.Combine(folder, "book.html"); + File.WriteAllText(source, html, new UTF8Encoding(false)); + var document = HtmlReader.Read(source, Path.Combine(folder, "media")); + var note = new HNote(endnote: false); + var noteParagraph = new HParagraph(); + noteParagraph.Inlines.Add(new HText("A note at the end.", default)); + note.Blocks.Add(noteParagraph); + document.Sections[^1].Blocks.OfType().Last().Inlines.Add(note); + var metadata = new BookMetadata { Title = Title, Language = "en", Publisher = "Harbour Press", Description = "A short story." }; + metadata.Authors.Add(Author); + return new Book(document, metadata); + } + + private static string Text(HDocument document) => PlainTextWriter.Write(document); + + private static IEnumerable Inlines(HDocument document) => document.Sections.SelectMany(s => HDocumentWalker.Inlines(s.Blocks)); + + private static List Headings(HDocument document, int level) => + [.. document.Sections.SelectMany(s => s.Blocks).OfType().Where(p => p.HeadingLevel == level).Select(p => HDocumentWalker.Text(p.Inlines))]; + + private string Output(string name) => Path.Combine(fx.NewFolder(), name); + + /// Hands out paths in one folder, as the job queue does for the final output. + private sealed class FolderAllocator(string folder, string name) : IOutputAllocator + { + public string? Allocate(string extension, string? suffix = null) => + Path.Combine(folder, name + suffix + (extension.Length > 0 ? "." + extension : "")); + } + + // ───────────────────────── Routes ───────────────────────── + + [Fact] + public void Legacy_formats_are_written_without_calibre_and_calibre_ranked_first_takes_epub() + { + var builtIn = new ConverterCatalog(fx.Converters.Where(c => !OptionalEngines.Contains(c.Id))) { Priority = fx.Catalog.Priority }.CreatePlanner(["rhwp"]); + foreach (var to in EbookConverter.LegacyWriters) + foreach (var from in new[] { "epub", "mobi", "azw3", "fb2", "htmlz", "chm", "pdb", "pml", "tcr", "rb", "snb", "oeb", "md", "html", "txt", "docx" }) + if (from != to) + Assert.Equal("ebook", Assert.Single(builtIn.Plan(from, to)!.Steps).Converter.Id); + + var withCalibre = fx.Catalog.CreatePlanner(["calibre", "rhwp"]); + foreach (var to in EbookConverter.LegacyWriters) + Assert.Equal("ebook", Assert.Single(withCalibre.Plan("epub", to)!.Steps).Converter.Id); + + var order = fx.Catalog.Priority.Where(id => id != "calibre").Prepend("calibre").ToList(); + var calibreFirst = new ConverterCatalog(fx.Converters) { Priority = order }.CreatePlanner(["calibre"]); + foreach (var to in EbookConverter.LegacyWriters) + Assert.Equal("calibre", Assert.Single(calibreFirst.Plan("epub", to)!.Steps).Converter.Id); + } + + [Theory] + [InlineData("pdb")] + [InlineData("pml")] + [InlineData("tcr")] + [InlineData("rb")] + [InlineData("snb")] + public async Task Epub_converts_to_each_format_and_reads_back(string format) + { + var epub = EbookBuilders.Epub(fx.NewFolder()); + await using var queue = new JobQueue(new ConverterCatalog(fx.Converters.Where(c => !OptionalEngines.Contains(c.Id))) { Priority = fx.Catalog.Priority }); + var job = new ConversionJob([epub], new Preset { TargetFormat = format }, format); + + await queue.RunAsync(job); + + Assert.True(job.State == JobState.Completed, string.Join("; ", job.Files.Select(f => $"{f.ErrorKey} {f.ErrorDetail}"))); + var output = Assert.Single(job.Outputs); + Assert.EndsWith("." + format, output); + var book = format switch + { + "pdb" => PdbReader.ReadBook(output, fx.NewFolder(), Token), + "pml" => PmlReader.ReadBook(output, fx.NewFolder(), Token), + "tcr" => TcrReader.ReadBook(output, fx.NewFolder(), Token), + "rb" => RocketBookReader.ReadBook(output, fx.NewFolder(), Token), + _ => SnbReader.ReadBook(output, fx.NewFolder(), Token), + }; + var source = EpubReader.ReadBook(epub, fx.NewFolder(), Token).Document; + var expected = Headings(source, 1).Concat(Headings(source, 2)).Take(3); + var text = Text(book.Document); + foreach (var heading in expected) + Assert.Contains(heading, text); + if (format is "pml" or "rb" or "snb") + Assert.NotEmpty(Inlines(book.Document).OfType()); + if (format is "rb" or "snb") + Assert.Equal("별빛 이야기", book.Metadata.Title); + if (format == "pml") + Assert.True(Directory.Exists(Path.ChangeExtension(output, null) + PmlPictures.FolderSuffix), "the pictures sit beside the book"); + } + + // ───────────────────────── PDB ───────────────────────── + + [Fact] + public void Pdb_reads_back_with_its_text_title_and_bookmarks() + { + var book = SampleBook(); + var path = Output("book.pdb"); + PdbWriter.Write(book, path, path, Token); + + var read = PdbReader.ReadBook(path, fx.NewFolder(), Token); + Assert.Equal(Title, read.Metadata.Title); + var expected = LegacyWriting.PlainText(book.Document, "\n"); + var database = PalmDatabase.Open(File.ReadAllBytes(path)); + Assert.Equal(expected, Encoding.UTF8.GetString(MobiReader.PalmDocText(database))); + var text = Text(read.Document); + foreach (var part in new[] { "Chapter One", Accented, Korean, "First item", Line1, "A note at the end." }) + Assert.Contains(part, text); + + Assert.Equal("TEXtREAd", database.TypeCreator); + var header = database.Record(0); + Assert.Equal(16, header.Length); + Assert.Equal(2, PalmDatabase.U16(header, 0)); // PalmDOC compression + var length = Encoding.UTF8.GetByteCount(expected); + Assert.Equal(length, PalmDatabase.U32(header, 4)); + var textRecords = (length + 4095) / 4096; + Assert.Equal(textRecords, PalmDatabase.U16(header, 8)); + Assert.Equal(4096, PalmDatabase.U16(header, 10)); + // Bookmarks after the text: Chapter One, A Section, Chapter Two, each at its heading's byte offset. + Assert.Equal(1 + textRecords + 3, database.Count); + var bytes = Encoding.UTF8.GetBytes(expected); + var names = new List(); + for (var i = 1 + textRecords; i < database.Count; i++) + { + var record = database.Record(i); + Assert.Equal(20, record.Length); + var name = Encoding.Latin1.GetString(record[..16]).TrimEnd('\0'); + names.Add(name); + var position = (int)PalmDatabase.U32(record, 16); + Assert.StartsWith(name, Encoding.UTF8.GetString(bytes, position, bytes.Length - position)); + } + Assert.Equal(["Chapter One", "A Section", "Chapter Two"], names); + } + + [Fact] + public void Pdb_and_tcr_text_is_windows_1252_when_it_fits_and_utf8_otherwise() + { + Encoding.RegisterProvider(CodePagesEncodingProvider.Instance); + var latin = new Book(PlainText.ToDocument($"{Accented}\n\n{Line1}\n"), new BookMetadata { Title = Title }); + var pdb = Output("latin.pdb"); + PdbWriter.Write(latin, pdb, pdb, Token); + Assert.Equal(Encoding.GetEncoding(1252).GetBytes($"{Accented}\n\n{Line1}\n"), MobiReader.PalmDocText(PalmDatabase.Open(File.ReadAllBytes(pdb)))); + Assert.Contains(Accented, Text(PdbReader.ReadBook(pdb, fx.NewFolder(), Token).Document)); + + var tcr = Output("latin.tcr"); + TcrWriter.Write(latin, tcr, Token); + Assert.Equal(Encoding.GetEncoding(1252).GetBytes($"{Accented}\r\n\r\n{Line1}\r\n"), TcrReader.Decompress(File.ReadAllBytes(tcr), Token)); + + var korean = SampleBook(); + TcrWriter.Write(korean, tcr, Token); + Assert.Equal(Encoding.UTF8.GetBytes(LegacyWriting.PlainText(korean.Document, "\r\n")), TcrReader.Decompress(File.ReadAllBytes(tcr), Token)); + Assert.Contains(Korean, Text(TcrReader.ReadBook(tcr, fx.NewFolder(), Token).Document)); + } + + // ───────────────────────── TCR ───────────────────────── + + [Fact] + public void Tcr_reads_back_exactly_with_a_dictionary_that_compresses() + { + var book = SampleBook(); + var path = Output("book.tcr"); + TcrWriter.Write(book, path, Token); + + var read = TcrReader.ReadBook(path, fx.NewFolder(), Token); + foreach (var part in new[] { "Chapter Two", Accented, Korean, Line1 }) + Assert.Contains(part, Text(read.Document)); + + // A long text: every byte it uses has an entry, all 256 codes are used, and it shrinks well. + var story = Encoding.UTF8.GetBytes(string.Concat(Enumerable.Range(1, 3000).Select(i => $"Line {i}: the keeper climbed the stairs and trimmed the wick. 등대 {i % 7}\r\n"))); + var tcr = TcrWriter.Compress(story, Token); + Assert.Equal(story, TcrReader.Decompress(tcr, Token)); + Assert.True(tcr.AsSpan().StartsWith("!!8-Bit!!"u8)); + var entries = TcrWriter.Dictionary(story, Token); + Assert.Equal(256, entries.Count); + Assert.All(story.Distinct(), b => Assert.Contains(entries, e => e is [var only] && only == b)); + Assert.All(entries, e => Assert.InRange(e.Length, 1, 255)); + Assert.True(tcr.Length < story.Length / 2, $"{tcr.Length} of {story.Length} bytes"); + + // Every byte value: no code is left for pairs, and the text still comes back. + var all = Enumerable.Range(0, 512).Select(i => (byte)i).ToArray(); + Assert.Equal(all, TcrReader.Decompress(TcrWriter.Compress(all, Token), Token)); + Assert.Equal("!!8-Bit!!"u8.ToArray().Concat(Enumerable.Repeat((byte)0, 256)), TcrWriter.Compress([], Token)); + } + + // ───────────────────────── PML ───────────────────────── + + [Fact] + public void Pml_keeps_chapters_styles_links_notes_pictures_and_characters() + { + var book = SampleBook(); + var folder = fx.NewFolder(); + var path = Path.Combine(folder, "book.pml"); + PmlWriter.Write(book, path, new PmlPictures(path, new FolderAllocator(folder, "book"), fx.NewFolder()), Token); + + var bytes = File.ReadAllBytes(path); + Assert.All(bytes, b => Assert.InRange(b, (byte)0x0A, (byte)0x7E)); // ASCII only + var pml = Encoding.ASCII.GetString(bytes); + Assert.Contains("\\xChapter One\\x\r\n", pml); + Assert.Contains("\\X1A Section\\X1\r\n", pml); + Assert.Contains("\\Blit\\B", pml); + Assert.Contains("\\idusk\\i", pml); + Assert.Contains("Caf\\a233, na\\a239ve \\a151 \\a147quoted\\a148 \\a1285", pml); + Assert.Contains("\\UB4F1", pml); // 등 + Assert.Contains("Back\\\\slash and \\a060footnote>", pml); + Assert.Contains("\\q=\"#two\"the second chapter\\q", pml); + Assert.Contains("\\Q=\"two\"", pml); + Assert.Contains("\\m=\"img0001.png\"", pml); + Assert.Contains("H\\Sb2\\SbO", pml); + Assert.Contains("\\Fn=\"fn1\"1\\Fn", pml); + Assert.Contains("\r\nA note at the end.", pml); + var picture = Path.Combine(folder, "book" + PmlPictures.FolderSuffix, "img0001.png"); + using (var image = new MagickImage(picture)) + Assert.Equal(MagickFormat.Png, image.Format); + + var read = PmlReader.ReadBook(path, fx.NewFolder(), Token).Document; + Assert.Equal(["Chapter One", "Chapter Two"], Headings(read, 1)); + Assert.Equal(["A Section"], Headings(read, 2)); + var text = Text(read); + foreach (var part in new[] { Accented, Korean, "Back\\slash and text.", "● First item", "Name | Value", Line1, "A note at the end." }) + Assert.Contains(part, text); + var inlines = Inlines(read).ToList(); + Assert.Contains(inlines, i => i is HText { Text: "lit", Format.Bold: true }); + Assert.Contains(inlines, i => i is HText { Text: "dusk", Format.Italic: true }); + Assert.Contains(inlines, i => i is HText { Text: "2", Format.Subscript: true }); + Assert.Single(inlines.OfType()); + var link = Assert.Single(inlines.OfType(), l => HDocumentWalker.Text(l.Content) == "the second chapter"); + Assert.Contains(inlines, i => i is HBookmark b && "#" + b.Name == link.Target); + } + + [Fact] + public void Pml_without_pictures_writes_no_folder() + { + var folder = fx.NewFolder(); + var path = Path.Combine(folder, "book.pml"); + PmlWriter.Write(SampleBook(), path, pictures: null, Token); + + Assert.Equal([path], Directory.GetFileSystemEntries(folder)); + Assert.DoesNotContain("\\m=", File.ReadAllText(path)); + } + + // ───────────────────────── RB ───────────────────────── + + [Fact] + public void Rb_keeps_metadata_headings_styles_links_and_pictures() + { + var book = SampleBook(); + var path = Output("book.rb"); + RocketBookWriter.Write(book, path, path, fx.NewFolder(), Token); + + var data = File.ReadAllBytes(path); + Assert.True(data.AsSpan().StartsWith((ReadOnlySpan)[0xB0, 0x0C, 0xB0, 0x0C, 0x02, 0x00, (byte)'N', (byte)'U', (byte)'V', (byte)'O'])); + Assert.Equal(0x128u, BinaryPrimitives.ReadUInt32LittleEndian(data.AsSpan(0x18))); + Assert.Equal((uint)data.Length, BinaryPrimitives.ReadUInt32LittleEndian(data.AsSpan(0x1C))); + Assert.Equal(3u, BinaryPrimitives.ReadUInt32LittleEndian(data.AsSpan(0x128))); + var entries = Enumerable.Range(0, 3).Select(i => + { + var entry = data.AsSpan(0x128 + 4 + i * 44, 44); + return (Name: Encoding.ASCII.GetString(entry[..32]).TrimEnd('\0'), Length: BinaryPrimitives.ReadInt32LittleEndian(entry[32..]), + Offset: BinaryPrimitives.ReadInt32LittleEndian(entry[36..]), Flags: BinaryPrimitives.ReadInt32LittleEndian(entry[40..])); + }).ToList(); + Assert.Equal([("book.info", 8), ("book.html", 1), ("img0001.png", 0)], entries.Select(e => (e.Name, e.Flags))); + Assert.Equal(0x128 + 4 + 3 * 44, entries[0].Offset); + Assert.Equal(data.Length, entries[^1].Offset + entries[^1].Length); + var info = Encoding.Latin1.GetString(data, entries[0].Offset, entries[0].Length); + Assert.Equal($"TITLE={Title}\r\nAUTHOR={Author}\r\nBODY=book.html\r\n", info); + // The page: its chunk count fits its size (4 KB each), and every chunk is a zlib stream. + var page = data.AsSpan(entries[1].Offset, entries[1].Length); + var chunks = BinaryPrimitives.ReadInt32LittleEndian(page); + var size = BinaryPrimitives.ReadInt32LittleEndian(page[4..]); + Assert.Equal((size + 4095) / 4096, chunks); + Assert.Equal(0x78, page[8 + chunks * 4]); + + var read = RocketBookReader.ReadBook(path, fx.NewFolder(), Token); + Assert.Equal(Title, read.Metadata.Title); + Assert.Equal([Author], read.Metadata.Authors); + Assert.Equal(["Chapter One", "Chapter Two"], Headings(read.Document, 1)); + Assert.Equal(["A Section"], Headings(read.Document, 2)); + var text = Text(read.Document); + foreach (var part in new[] { Accented, Korean, "Back\\slash and text.", "First item", Line1, "A note at the end." }) + Assert.Contains(part, text); + var inlines = Inlines(read.Document).ToList(); + Assert.Contains(inlines, i => i is HText { Text: "lit", Format.Bold: true }); + Assert.Contains(inlines, i => i is HText { Text: "dusk", Format.Italic: true }); + Assert.Single(inlines.OfType()); + var link = Assert.Single(inlines.OfType(), l => HDocumentWalker.Text(l.Content) == "the second chapter"); + Assert.Contains(inlines, i => i is HBookmark b && "#" + b.Name == link.Target); + Assert.Contains(read.Document.Sections.SelectMany(s => s.Blocks), b => b is HTable); + } + + // ───────────────────────── SNB ───────────────────────── + + [Fact] + public void Snb_keeps_metadata_cover_chapters_and_pictures() + { + var book = SampleBook(); + var cover = Path.Combine(fx.NewFolder(), "cover.png"); + File.WriteAllBytes(cover, EbookBuilders.Png(30, 45, MagickColors.Navy)); + book.Metadata.CoverImage = cover; + var path = Output("book.snb"); + SnbWriter.Write(book, path, path, fx.NewFolder(), Token); + + var data = File.ReadAllBytes(path); + Assert.True(data.AsSpan().StartsWith("SNBP000B"u8)); + Assert.True(data.AsSpan(data.Length - 8).SequenceEqual("SNBP000B"u8)); + var files = SnbReader.Unpack(data, DocumentLimits.Default, Token); + Assert.Equal(BinaryPrimitives.ReadInt32BigEndian(data.AsSpan(20)), files.Count); + Assert.Equal(["snbc/ch0001.snbc", "snbc/ch0002.snbc", "snbc/images/img0001.png", "snbc/images/img0002.jpg", "snbf/book.snbf", "snbf/toc.snbf"], + files.Select(f => f.Name)); + var toc = Encoding.UTF8.GetString(files.Single(f => f.Name == "snbf/toc.snbf").Bytes); + Assert.Contains("2", toc); + Assert.Contains("Chapter Two", toc); + var chapter = Encoding.UTF8.GetString(files.Single(f => f.Name == "snbc/ch0001.snbc").Bytes); + Assert.Contains("img0002.jpg", chapter); + Assert.Contains("", chapter); + // The text stream's one block is bzip2. + var textSize = BinaryPrimitives.ReadInt32BigEndian(data.AsSpan(36)); + Assert.Equal(files.Where(f => f.Name.Contains(".snb")).Sum(f => f.Bytes.Length), textSize); + var binarySize = BinaryPrimitives.ReadInt32BigEndian(data.AsSpan(32)); + var textBlock = 44 + BinaryPrimitives.ReadInt32BigEndian(data.AsSpan(28)) + binarySize; + Assert.True(data.AsSpan(textBlock).StartsWith("BZh9"u8)); + + var read = SnbReader.ReadBook(path, fx.NewFolder(), Token); + Assert.Equal(Title, read.Metadata.Title); + Assert.Equal([Author], read.Metadata.Authors); + Assert.Equal("en", read.Metadata.Language); + Assert.Equal("Harbour Press", read.Metadata.Publisher); + Assert.Equal("A short story.", read.Metadata.Description); + Assert.NotNull(read.Metadata.CoverImage); + Assert.Equal(["Chapter One", "Chapter Two"], Headings(read.Document, 1)); + var text = Text(read.Document); + foreach (var part in new[] { Accented, Korean, "● First item", "Name | Value", Line1, "[1] A note at the end." }) + Assert.Contains(part, text); + Assert.Single(Inlines(read.Document).OfType()); + } + + [Fact] + public void Snb_text_of_several_blocks_reads_back_and_incompressible_blocks_are_stored() + { + // Text of several blocks: each one bzip2. + var story = Encoding.UTF8.GetBytes(string.Concat(Enumerable.Range(1, 4000).Select(i => $"

Line {i} of the story.

"))); + var data = SnbWriter.Container([new("snbc/a.snbc", story, Text: true)], Token); + Assert.Equal(story, Assert.Single(SnbReader.Unpack(data, DocumentLimits.Default, Token)).Bytes); + Assert.True(story.Length > 3 * 0x8000); + + // Random bytes: the full block is stored, the short last one that bzip2 can't shrink below a block is padded + // to a stored block of its own. + var noise = new byte[2 * 0x8000 - 10]; + new Random(7).NextBytes(noise); + data = SnbWriter.Container([new("snbc/b.snbc", noise, Text: true)], Token); + Assert.Equal(noise, Assert.Single(SnbReader.Unpack(data, DocumentLimits.Default, Token)).Bytes); + Assert.Equal(2 * 0x8000, BinaryPrimitives.ReadInt32BigEndian(data.AsSpan(36))); + var firstBlock = 44 + BinaryPrimitives.ReadInt32BigEndian(data.AsSpan(28)); + Assert.True(data.AsSpan(firstBlock, 0x8000).SequenceEqual(noise.AsSpan(0, 0x8000))); + } + + // ───────────────────────── Limits and failures ───────────────────────── + + [Fact] + public void Books_beyond_the_readers_limits_are_refused() + { + var text = Assert.Throws(() => LegacyWriting.CheckText(DocumentLimits.Default.MaxPartBytes + 1, "PDB")); + Assert.Equal(DocumentLimits.ErrorKey, text.ErrorKey); + var records = Assert.Throws(() => PalmDatabase.Build("x", "TEXtREAd", Enumerable.Repeat(Array.Empty(), 70_000).ToList())); + Assert.Equal(DocumentLimits.ErrorKey, records.ErrorKey); + var many = Enumerable.Range(0, DocumentLimits.Default.MaxPackageEntries + 1).Select(i => new SnbWriter.SnbFile($"f{i}", [], Text: true)).ToList(); + Assert.Equal(DocumentLimits.ErrorKey, Assert.Throws(() => SnbWriter.Container(many, Token)).ErrorKey); + } + + [Theory] + [InlineData("pdb")] + [InlineData("pml")] + [InlineData("tcr")] + [InlineData("rb")] + [InlineData("snb")] + public void A_cancelled_write_leaves_no_file(string format) + { + var book = SampleBook(); + var folder = fx.NewFolder(); + var path = Path.Combine(folder, "book." + format); + using var cancelled = new CancellationTokenSource(); + cancelled.Cancel(); + + Assert.ThrowsAny(() => + { + switch (format) + { + case "pdb": + PdbWriter.Write(book, path, path, cancelled.Token); + break; + case "pml": + PmlWriter.Write(book, path, new PmlPictures(path, new FolderAllocator(folder, "book"), fx.NewFolder()), cancelled.Token); + break; + case "tcr": + TcrWriter.Write(book, path, cancelled.Token); + break; + case "rb": + RocketBookWriter.Write(book, path, path, fx.NewFolder(), cancelled.Token); + break; + default: + SnbWriter.Write(book, path, path, fx.NewFolder(), cancelled.Token); + break; + } + }); + Assert.Empty(Directory.GetFiles(folder, "*", SearchOption.AllDirectories)); + } +} From bfb19a83220f2ca47395d535df13e5689933fb95 Mon Sep 17 00:00:00 2001 From: KnifeLemon Date: Fri, 20 Nov 2026 21:12:05 +0900 Subject: [PATCH 2/6] feat(markup): built-in LaTeX, AsciiDoc, Typst and man page writers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - LatexWriter, AsciiDocWriter, TypstWriter and ManWriter on the shared MarkupTextWriter walk: headings, emphasis, nested and numbered lists, quotes, code, tables with merged cells, links, footnotes, pictures, rules, and every special character of each language escaped. - MarkupConverter.WriteOnly offers every built-in reader's format into tex, adoc, typ and man at cost 8, below Pandoc's 11; reading them stays Pandoc's. Output goes through a temporary file; the walk checks the conversion's cancellation between blocks. - TexMathWriter writes the equation tree (Word math, 한글 equations, MathML) as TeX for LaTeX, AsciiDoc and Markdown, so Markdown output no longer writes linear text for them; TypstMathWriter writes Typst math. Trees deeper than 256 levels keep their text. - Tests build their models in code; the hostile writer tests cover the four new writers. Co-Authored-By: Claude Opus 5.5 --- docs/ENGINES.md | 50 +- src/Filee.Engines/Hwp/Hwpx/LatexMath.cs | 32 ++ src/Filee.Engines/Hwp/Hwpx/MarkdownWriter.cs | 15 +- .../Hwp/Hwpx/Markup/AsciiDocWriter.cs | 321 +++++++++++ .../Hwp/Hwpx/Markup/LatexWriter.cs | 441 +++++++++++++++ .../Hwp/Hwpx/Markup/ManWriter.cs | 237 ++++++++ .../Hwp/Hwpx/Markup/MarkupTableOwners.cs | 29 + .../Hwp/Hwpx/Markup/MarkupTextWriter.cs | 29 + .../Hwp/Hwpx/Markup/TypstWriter.cs | 280 ++++++++++ src/Filee.Engines/Hwp/Hwpx/TexMathWriter.cs | 516 ++++++++++++++++++ src/Filee.Engines/Hwp/Hwpx/TypstMathWriter.cs | 479 ++++++++++++++++ src/Filee.Engines/Text/MarkupConverter.cs | 40 +- .../MacroMarkupWriterTests.cs | 500 +++++++++++++++++ tests/Filee.Engines.Tests/MarkupRouteTests.cs | 6 +- .../MarkupWriterLimitTests.cs | 4 + .../Filee.Engines.Tests/PandocMarkupTests.cs | 10 +- .../Filee.Engines.Tests/TexMathWriterTests.cs | 194 +++++++ 17 files changed, 3162 insertions(+), 21 deletions(-) create mode 100644 src/Filee.Engines/Hwp/Hwpx/Markup/AsciiDocWriter.cs create mode 100644 src/Filee.Engines/Hwp/Hwpx/Markup/LatexWriter.cs create mode 100644 src/Filee.Engines/Hwp/Hwpx/Markup/ManWriter.cs create mode 100644 src/Filee.Engines/Hwp/Hwpx/Markup/MarkupTableOwners.cs create mode 100644 src/Filee.Engines/Hwp/Hwpx/Markup/TypstWriter.cs create mode 100644 src/Filee.Engines/Hwp/Hwpx/TexMathWriter.cs create mode 100644 src/Filee.Engines/Hwp/Hwpx/TypstMathWriter.cs create mode 100644 tests/Filee.Engines.Tests/MacroMarkupWriterTests.cs create mode 100644 tests/Filee.Engines.Tests/TexMathWriterTests.cs diff --git a/docs/ENGINES.md b/docs/ENGINES.md index 3ed3dc706..2a925fda5 100644 --- a/docs/ENGINES.md +++ b/docs/ENGINES.md @@ -848,7 +848,9 @@ for dollars: no space inside the delimiters, no digit after the closing `$`, cod equations, → HTML and EPUB MathML (with the TeX as its annotation). Display math alone in its paragraph is centred. An unknown command stays as its text (`\foo`); a formula longer than 32,768 characters or nested deeper than 100 levels stays its source, and the equations count against `MaxObjects`. Writers without equations keep the TeX -source as written (`$\frac{a}{b}$`). +source as written (`$\frac{a}{b}$`). The other way, Word math, 한글 equations and MathML become TeX in Markdown, +LaTeX and AsciiDoc (`Hwp/Hwpx/TexMathWriter.cs`) and Typst math in Typst (`TypstMathWriter.cs`), so DOCX or HWP → +Markdown keeps them as equations. ## Spreadsheets @@ -4152,8 +4154,9 @@ Filee renderer take it: → HWPX, DOCX, ODT, PDF), and a writer from the model. `markup`) offers every built-in reader's format → each of these languages, and each of them → Markdown (`MarkdownWriter`, pictures in a `_files` folder), HTML (one page, pictures embedded) and TXT. Its edges cost 8 like the DOCX writer's, below Pandoc's (10 out of a markup language, 11 into one), so the built-in route wins -wherever both exist. Pandoc keeps what only it does: LaTeX, Typst, AsciiDoc and man pages (macro and scripting -languages), slides (PPTX), and RTF and EPUB written straight from a markup language. +wherever both exist. LaTeX, AsciiDoc, Typst and man pages (macro and scripting languages) are written here too, from +every built-in reader's format ([below](#latex-asciidoc-typst-and-man-pages)); Pandoc keeps reading them, slides +(PPTX), and RTF and EPUB written straight from a markup language. What the readers share (`MarkupBlocks`): code blocks are shaded paragraphs as the Markdown and HTML readers make them (a blank line inside one is an empty shaded run, so a block stays one block), quotes are indented 2000 HWPUNIT per @@ -4297,6 +4300,44 @@ inlinemediaobject pictures (linked into `_files`), footnotes, links, ancho HWPX / PDF / HTML / Markdown with Pandoc absent and the hostile inputs (DTDs and entity expansion, nesting, a million openers, wide and long tables); `DocumentFuzzTests` mutates each. +### LaTeX, AsciiDoc, Typst and man pages + +Written, not read (`MarkupConverter.WriteOnly`, cost 8 like the other writers, below Pandoc's 11): `LatexWriter`, +`AsciiDocWriter`, `TypstWriter` and `ManWriter` in `Hwp/Hwpx/Markup/`, on the walk the other text markup writers +share (`MarkupTextWriter`: `Nesting.Limit` first, the table grid and text caps of `MarkupWriteBudget`, cancellation +between blocks). The result is written through a temporary file beside it. + +- **LaTeX**: a whole article for pdfLaTeX, XeLaTeX or LuaLaTeX (`iftex`: fontenc / inputenc or fontspec; xeCJK or + luatexja when there is CJK text, which needs XeLaTeX or LuaLaTeX). The preamble loads only what the body uses: + amsmath / amssymb, graphicx (pictures no wider than the text), longtable / array / multirow, ulem, hyperref. + Unnumbered `\section` … `\subparagraph`, `quote`, itemize / enumerate to LaTeX's four levels (numbers as item + labels), `verbatim` (typewriter lines in table cells or when the code holds `\end{verbatim}`), longtable (tabular + in a cell) with p columns, `\multicolumn`, `\multirow` and `\cline` rules that leave merged cells open, header + rows repeated, `\href`, `\includegraphics`, `\footnote`. Every special character is escaped (`\ { } $ & # _ % ~ + ^`, brackets braced so `\` and `\item` don't take them as options, `< > | "` and backticks, the `--` `''` + ligatures). +- **AsciiDoc** (Asciidoctor): `= Title`, `:stem: latexmath` when there are equations, `==` sections, `+` line + breaks, `____` quotes, `*` / `.` lists (`[start=N]`), `----` listings (callouts and preprocessor directives + defused), `|===` tables with `2.3+|` spans and `a|` cells holding blocks (one nested `!===` level), roles for + underline and strike, `link:++…++[…]`, `image:`, `footnote:[…]`. Text that would format, substitute, link or start a + block goes into `++passthroughs++`, `{plus}` / `{vbar}`, or gets `{empty}` at the line start. +- **Typst** (0.11 or later): `#set document`, `=` headings, `\` line breaks, `#quote`, `-` / `N.` lists, raw blocks, + `#table` with `table.header` and `table.cell(colspan:, rowspan:)`, `#strong` `#emph` `#underline` `#strike` + `#super` `#sub` `#highlight`, `#link`, `#image`, `#footnote`; every markup character escaped. +- **man**: `.TH`, `.SH` / `.SS`, `.PP`, `.RS` / `.RE`, `.IP` items, `.EX` / `.EE`, tbl tables (`s` and `^` for + merged cells, `T{ … T}` cells), fonts `\fB \fI \f(CR`, links as text and ``, footnotes under NOTES. ASCII + only (`\e`, `\-`, `\(dq`, `\(aq`, `\[uXXXX]`); a line that would be a request gets `\&`. No pictures. +- Equations: TeX (`$…$`, `\[…\]`; a source that is a display environment of its own as it is) in LaTeX, Markdown and + AsciiDoc, from the equation's TeX source or written from its tree by `Hwp/Hwpx/TexMathWriter.cs` (Word math, 한글 + equations, MathML: symbols by the commands `LatexMath` reads, amsmath environments, `\text` for words); + `Hwp/Hwpx/TypstMathWriter.cs` writes Typst's own math (`frac(a, b)`, `x_(i)^(2)`, `mat(…)`, `cases(…)`, Unicode + symbols, escaped syntax). man pages get the linear text. A tree deeper than 256 levels keeps its text. + +`MacroMarkupWriterTests` builds one model in code with all of the above and checks each language's text and +structure (balanced braces and environments, requests, brackets), the routes and cancellation; `TexMathWriterTests` +reads TeX, writes it back and reads it again to the same tree, and reads Markdown written from Word and 한글 equations +back as the same equations; `MarkupWriterLimitTests` puts the four writers through its hostile models. + ### Mind maps `MindMapReader` (#1543, #1004) reads FreeMind / Freeplane (`.mm`) and XMind (`.xmind`) maps as an **outline**: the @@ -4338,7 +4379,8 @@ many topics, an oversized part, a ZIP bomb); `MindMapFuzzTests` mutates the map, ## Pandoc `Office/PandocConverter.cs` runs Pandoc for the markup languages Filee doesn't read itself: LaTeX, AsciiDoc -(`.adoc`, `.asciidoc`), Typst (`.typ`) and man pages (`.man`) — macro and scripting languages. Pandoc 3.12 also +(`.adoc`, `.asciidoc`), Typst (`.typ`) and man pages (`.man`) — macro and scripting languages (Filee writes them +itself, [above](#latex-asciidoc-typst-and-man-pages): Pandoc reads them and converts between them). Pandoc 3.12 also reads and writes reStructuredText, Org (`.org`), MediaWiki (`.mediawiki`, `.wiki`), DocBook (`.dbk`, `.docbook`; written as DocBook 5), Jupyter notebooks (`.ipynb`), Textile (`.textile`) and OPML (`.opml`), but Filee reads and writes those itself ([above](#lightweight-markup-languages)) and its routes win; Pandoc's stay for RTF and EPUB diff --git a/src/Filee.Engines/Hwp/Hwpx/LatexMath.cs b/src/Filee.Engines/Hwp/Hwpx/LatexMath.cs index f9b766120..829f23ad2 100644 --- a/src/Filee.Engines/Hwp/Hwpx/LatexMath.cs +++ b/src/Filee.Engines/Hwp/Hwpx/LatexMath.cs @@ -118,6 +118,21 @@ void Add(Kind kind, string list) private static readonly Dictionary BigGlyphs = BigOperators.Values.DistinctBy(v => v.Glyph).ToDictionary(v => v.Glyph, v => v.Command); + /// + /// Symbol characters → the first command this reader takes for them, for TexMathWriter. Commands that need more + /// than amsmath and amssymb (or don't exist in LaTeX) are left out: the writer has its own way for those. + /// + private static readonly Dictionary SymbolNames = Symbols + .Where(s => s.Key is not ("omicron" or "degree" or "iddots" or "copyright" or "euro" or "coloneqq" or "eqqcolon" or "neqsim" or "nequiv")) + .DistinctBy(s => s.Value.Glyph).ToDictionary(s => s.Value.Glyph, s => s.Key, StringComparer.Ordinal); + + /// The TeX command (without its backslash) of a symbol character: ≤ → leq, α → alpha; null for none. + internal static string? SymbolCommand(string glyph) => SymbolNames.GetValueOrDefault(glyph); + + /// The alphabet command and letter of a math alphabet letter (ℝ → mathbb, R; 𝓛 → mathcal, L), or null. + internal static (string Command, char Letter)? AlphabetLetter(string glyph) => + Reader.AlphabetLetters.TryGetValue(glyph, out var letter) ? letter : null; + /// Function names 한글 and rhwp know (written upright by themselves). private static readonly HashSet Functions = new(StringComparer.Ordinal) { @@ -998,6 +1013,23 @@ private enum Alphabets Fraktur, } + /// Each letter of the math alphabets → the command and the letter it is read from (ℝ → mathbb, R). + public static readonly Dictionary AlphabetLetters = BuildAlphabetLetters(); + + private static Dictionary BuildAlphabetLetters() + { + var letters = new Dictionary(StringComparer.Ordinal); + foreach (var (alphabet, command) in new[] { (Alphabets.DoubleStruck, "mathbb"), (Alphabets.Script, "mathcal"), (Alphabets.Fraktur, "mathfrak") }) + { + for (var c = 'A'; c <= 'Z'; c++) + { + letters.TryAdd(Letter(c, alphabet), (command, c)); + letters.TryAdd(Letter(char.ToLowerInvariant(c), alphabet), (command, char.ToLowerInvariant(c))); + } + } + return letters; + } + /// \mathbb{R}, \mathcal{L}, \mathfrak{g}: the letters of the Unicode math alphabet. private static MathNode Alphabet(string raw, Alphabets alphabet) { diff --git a/src/Filee.Engines/Hwp/Hwpx/MarkdownWriter.cs b/src/Filee.Engines/Hwp/Hwpx/MarkdownWriter.cs index da77087b9..eac8bcb52 100644 --- a/src/Filee.Engines/Hwp/Hwpx/MarkdownWriter.cs +++ b/src/Filee.Engines/Hwp/Hwpx/MarkdownWriter.cs @@ -1,6 +1,7 @@ // HDocument → Markdown (the dialect MarkdownReader reads back: GitHub tables, ~sub~, ^sup^, footnotes): headings, -// emphasis, code, lists, block quotes, pipe tables (with column alignment), links, pictures and footnotes. Used for -// e-books, PDF and HWP / HWPX → MD, and TXTZ. +// emphasis, code, lists, block quotes, pipe tables (with column alignment), links, pictures, footnotes and equations +// ($…$ and $$…$$: TeX math as it was read, or written from the equation tree of Word math, 한글 equations and MathML). +// Used for e-books, PDF and HWP / HWPX → MD, and TXTZ. using System.Text; @@ -13,6 +14,9 @@ internal sealed class MarkdownWriter private readonly ListNumbers _numbers = new(); private readonly Markup.MarkupWriteBudget _budget; + /// The display equation the paragraph being written holds alone, written as $$…$$. + private HEquation? _display; + private MarkdownWriter(Func imageTarget, Markup.MarkupWriteBudget? budget) { _imageTarget = imageTarget; @@ -74,7 +78,10 @@ private static bool IsCode(HParagraph paragraph) => private void Paragraph(HParagraph paragraph, StringBuilder sb, string prefix, HBlock? next) { + _display = paragraph.Inlines.Count(i => i is not HBookmark) == 1 && paragraph.Inlines.OfType().FirstOrDefault() is { } lone + && HEquation.IsDisplay(lone, paragraph) ? lone : null; var text = Inlines(paragraph.Inlines); + _display = null; if (paragraph.Inlines.Any(i => i is HShape { Kind: HShapeKind.Line }) && text.Trim().Length == 0) { sb.Append(prefix).Append("---\n\n"); @@ -211,6 +218,10 @@ private string Inlines(List inlines) case HEquation { Tex: not null } equation: sb.Append(equation.Text); // TeX math as it was written ($x$, \[x\]), not escaped break; + case HEquation equation when TexMathWriter.Write(equation.Math) is { Length: > 0 } tex: + // Word math, 한글 equations, MathML: TeX math too, which MarkdownReader reads back as equations. + sb.Append(equation.Display || _display == equation ? $"$${tex}$$" : $"${tex}$"); + break; case HText text: sb.Append(Run(text.Text, text.Format)); break; diff --git a/src/Filee.Engines/Hwp/Hwpx/Markup/AsciiDocWriter.cs b/src/Filee.Engines/Hwp/Hwpx/Markup/AsciiDocWriter.cs new file mode 100644 index 000000000..39e034146 --- /dev/null +++ b/src/Filee.Engines/Hwp/Hwpx/Markup/AsciiDocWriter.cs @@ -0,0 +1,321 @@ +// HDocument → AsciiDoc (as Asciidoctor reads it): "= Title" and a :stem: latexmath header when there are equations, +// == section titles (discrete ones in table cells), paragraphs with " +" at line breaks, ____ quote blocks (longer +// delimiters for deeper quotes), * / . lists nested by repeating the marker ("+" for an item's further paragraphs, +// [start=N] for a list that doesn't start at 1), ---- listing blocks, |=== tables with header rows, colspan / rowspan +// specifiers (2.3+|) and AsciiDoc cells (a|) for cells holding lists, code or tables (!=== inside them), ''' rules, +// **bold** __italic__ ``code`` ##mark## ^sup^ ~sub~ and [.underline] / [.line-through] roles, link:++url++[text], +// image: pictures (files in "_files" next to the result), footnote:[…] notes, and equations as latexmath:[…] or +// [latexmath] blocks (their TeX source, or TeX written from the equation tree). +// +// Escaping: the characters that would format, substitute or link (* _ ` # ^ ~ [ ] { } < > \ @ &, and : ; - . ( where +// they would start a URL, a description list, a dash, an ellipsis or a © replacement) are written in ++passthroughs++ +// (which still escape < > &), + and | as {plus} and {vbar}, a line starting with anything but a letter or digit gets +// {empty} in front, and a code line that is a preprocessor directive (include::, ifdef:: …) a backslash. + +using System.Globalization; +using System.Text; +using System.Text.RegularExpressions; + +namespace Filee.Engines.Hwp.Hwpx.Markup; + +/// Writes the document model as AsciiDoc. +internal sealed partial class AsciiDocWriter(Func pictures) : MarkupTextWriter(pictures) +{ + /// Section levels Asciidoctor has (== to ======). + private const int MaxSectionLevel = 5; + + /// List levels (* to *****). + private const int MaxListLevel = 4; + + /// Table cells around the text being written: one level can hold a table (!===), deeper ones only text. + private int _cell; + + public static string Write(HDocument document, Func pictures) => new AsciiDocWriter(pictures).Write(document); + + protected override void Start(StringBuilder sb, string? title) + { + var equations = Source is not null && HDocumentWalker.Inlines(Source.Sections.SelectMany(s => s.Blocks)).Any(i => i is HEquation); + if (title is not null) + sb.Append("= ").Append(Escape(title.Replace('\n', ' '))).Append('\n'); + if (equations) + sb.Append(":stem: latexmath\n"); + if (title is not null || equations) + sb.Append('\n'); + } + + protected override void End(StringBuilder sb) + { + } + + protected override void Heading(StringBuilder sb, int level, string text) + { + if (text.Length == 0) + return; + if (_cell > 0) + sb.Append("[discrete]\n"); + sb.Append('=', Math.Min(level, MaxSectionLevel) + 1).Append(' ').Append(text).Append("\n\n"); + } + + protected override void Paragraph(StringBuilder sb, string text, int quote) => + Quoted(sb, quote, () => sb.Append(Lines(text)).Append("\n\n")); + + private static void Quoted(StringBuilder sb, int quote, Action body) + { + for (var q = 0; q < quote; q++) + sb.Append('_', 4 + q).Append('\n'); + body(); + for (var q = quote - 1; q >= 0; q--) + sb.Append('_', 4 + q).Append('\n'); + if (quote > 0) + sb.Append('\n'); + } + + /// + /// A paragraph's lines, " +" ending each that breaks, each guarded against reading as a block (an empty one holds + /// {empty}, as an empty line would end the paragraph). + /// + private static string Lines(string text) => + string.Join(" +\n", text.Trim('\n').Split('\n').Select(line => line.Trim().Length == 0 ? "{empty}" : LineStart(line.Trim()))); + + protected override void List(StringBuilder sb, IReadOnlyList items, int quote) + { + var first = true; + foreach (var item in items) + { + if (item.Continuation) + { + if (!first) + sb.Append("+\n"); + sb.Append(Lines(item.Text)).Append('\n'); + continue; + } + if (first && item.Ordered && item.Number != 1) + sb.Append(CultureInfo.InvariantCulture, $"[start={item.Number}]\n"); + first = false; + var level = Math.Min(item.Level, MaxListLevel); + sb.Append(item.Ordered ? '.' : '*', level + 1).Append(' ').Append(Lines(item.Text)).Append('\n'); + } + sb.Append('\n'); + } + + protected override void Code(StringBuilder sb, IReadOnlyList lines, int quote) + { + var longest = lines.Where(l => l.Length >= 4 && l.All(c => c == '-')).Select(l => l.Length).DefaultIfEmpty(3).Max(); + var fence = new string('-', longest + 1); + Quoted(sb, quote, () => + { + // Callouts (<1> at a line's end) would be read as such: special characters only. + if (lines.Any(l => Callout().IsMatch(l))) + sb.Append("[subs=\"specialchars\"]\n"); + sb.Append(fence).Append('\n'); + foreach (var line in lines) + sb.Append(Directive().IsMatch(line) ? "\\" : "").Append(line).Append('\n'); + sb.Append(fence).Append("\n\n"); + }); + } + + protected override void Table(StringBuilder sb, MarkupTable table) + { + if (_cell > 1) + { + // Asciidoctor nests tables one level: deeper ones are their cells' text. + foreach (var row in table.Rows) + foreach (var cell in row.OfType()) + if (Content(cell, separator: '|') is { Length: > 0 } text) + sb.Append(text).Append("\n\n"); + return; + } + var separator = _cell == 0 ? '|' : '!'; + var owners = MarkupTableOwners.Owners(table); + sb.Append(CultureInfo.InvariantCulture, $"[cols=\"{table.Columns}*\""); + if (table.HeaderRows.Count > 0 && table.HeaderRows[0]) + sb.Append(",options=\"header\""); + sb.Append("]\n").Append(separator).Append("===\n"); + for (var r = 0; r < table.Rows.Count; r++) + { + for (var c = 0; c < table.Columns; c++) + { + var (ownerRow, ownerColumn) = owners[r, c]; + if (ownerRow >= 0 && (ownerRow, ownerColumn) != (r, c)) + continue; // covered by a merged cell + if (table.Rows[r][c] is not { } cell) + { + sb.Append(separator).Append('\n'); + continue; + } + if (cell.ColSpan > 1) + sb.Append(cell.ColSpan.ToString(CultureInfo.InvariantCulture)); + if (cell.RowSpan > 1) + sb.Append('.').Append(cell.RowSpan.ToString(CultureInfo.InvariantCulture)); + if (cell.ColSpan > 1 || cell.RowSpan > 1) + sb.Append('+'); + if (!cell.Simple) + sb.Append('a'); + sb.Append(separator).Append(' ').Append(Content(cell, separator)).Append('\n'); + } + sb.Append('\n'); + } + sb.Append(separator).Append("===\n\n"); + } + + /// A cell's content: its text with " +" at line breaks, or (an AsciiDoc cell) its blocks. + private string Content(MarkupCell cell, char separator) + { + string text; + if (cell.Simple) + { + text = Lines(cell.Text); + } + else + { + var blocks = new StringBuilder(); + _cell++; + try + { + Blocks(cell.Source.Blocks, blocks); + } + finally + { + _cell--; + } + text = blocks.ToString().Trim('\n'); + } + // A separator in the text would end the cell. + return separator == '!' ? text.Replace("!", "\\!", StringComparison.Ordinal) : text; + } + + protected override void Rule(StringBuilder sb) => sb.Append("'''\n\n"); + + protected override string Run(string text, HCharFormat format) + { + if (format.Shade == HtmlReader.CodeShade) + return Around(text, core => $"``{Escape(core)}``"); + return Around(Escape(text), core => + { + if (format.Superscript is true) + core = $"^{core.Replace(" ", "{sp}", StringComparison.Ordinal)}^"; + else if (format.Subscript is true) + core = $"~{core.Replace(" ", "{sp}", StringComparison.Ordinal)}~"; + if (format.Underline is true) + core = $"[.underline]##{core}##"; + if (format.Strike is true) + core = $"[.line-through]##{core}##"; + if (format.Shade == HtmlReader.MarkShade) + core = $"##{core}##"; + if (format.Italic is true) + core = $"__{core}__"; + if (format.Bold is true) + core = $"**{core}**"; + return core; + }); + } + + protected override string Link(string target, string content) => + $"link:++{target.Replace("+", "%2B", StringComparison.Ordinal).Replace(" ", "%20", StringComparison.Ordinal)}++[{content}]"; + + protected override string Picture(string link) => $"image:{link}[]"; + + protected override string LineBreak() => "\n"; + + protected override string NoteReference(int number) => $"footnote:[{Notes[number - 1]}]"; + + protected override string? Equation(HEquation equation) + { + if (Tex(equation) is not { } tex) + return null; + return $"latexmath:[{tex.Replace("]", "\\]", StringComparison.Ordinal)}]"; + } + + protected override bool DisplayEquation(StringBuilder sb, HEquation equation, int quote) + { + if (Tex(equation) is not { } tex) + return false; + Quoted(sb, quote, () => sb.Append("[latexmath]\n++++\n").Append(tex).Append("\n++++\n\n")); + return true; + } + + /// An equation's TeX: its source, else written from its tree; null when neither can be had. + private static string? Tex(HEquation equation) => + equation.Tex is { } source && source.Trim().Length > 0 ? source.Trim() : TexMathWriter.Write(equation.Math); + + // ───────────────────────── Escaping ───────────────────────── + + /// + /// Text with what Asciidoctor would read as formatting, an attribute, a macro, a link or a replacement put in + /// passthroughs (consecutive ones together), and + and | as their attributes. + /// + private static string Escape(string text) + { + var sb = new StringBuilder(text.Length + 16); + var pass = new StringBuilder(); + void Flush() + { + if (pass.Length > 0) + sb.Append("++").Append(pass).Append("++"); + pass.Clear(); + } + + for (var i = 0; i < text.Length; i++) + { + var ch = text[i]; + var next = i + 1 < text.Length ? text[i + 1] : '\0'; + var special = ch switch + { + '*' or '_' or '`' or '#' or '^' or '~' or '[' or ']' or '{' or '}' or '<' or '>' or '\\' or '@' or '&' => true, + ':' => next is ':' or '/', + ';' => next == ';', + '-' => next == '-', + '.' => next == '.' && i + 2 < text.Length && text[i + 2] == '.', + '(' => Replacement().IsMatch(text.AsSpan(i)), + _ => false, + }; + if (special) + { + pass.Append(ch); + continue; + } + Flush(); + switch (ch) + { + case '+': + sb.Append("{plus}"); + break; + case '|': + sb.Append("{vbar}"); + break; + case '\t': + sb.Append(' '); + break; + case '\n': + sb.Append('\n'); + break; + case < ' ' or '': + break; + default: + sb.Append(ch); + break; + } + } + Flush(); + return sb.ToString(); + } + + /// A line that doesn't start with a letter or digit gets {empty} first, so it can't start a block. + private static string LineStart(string line) => + line.Length > 0 && !char.IsLetterOrDigit(line[0]) && !line.StartsWith("{empty}", StringComparison.Ordinal) ? "{empty}" + line + : NumberedLine().IsMatch(line) ? "{empty}" + line + : line; + + /// "1. " or "a) " and the like: an ordered list item. + [GeneratedRegex(@"^[0-9a-zA-Z]{1,9}[.)](?:\s|$)")] + private static partial Regex NumberedLine(); + + [GeneratedRegex(@"^\((?:C|R|TM)\)")] + private static partial Regex Replacement(); + + [GeneratedRegex(@"\s*$")] + private static partial Regex Callout(); + + [GeneratedRegex(@"^(?:include|ifdef|ifndef|ifeval|endif)::")] + private static partial Regex Directive(); +} diff --git a/src/Filee.Engines/Hwp/Hwpx/Markup/LatexWriter.cs b/src/Filee.Engines/Hwp/Hwpx/Markup/LatexWriter.cs new file mode 100644 index 000000000..98fc5b8f9 --- /dev/null +++ b/src/Filee.Engines/Hwp/Hwpx/Markup/LatexWriter.cs @@ -0,0 +1,441 @@ +// HDocument → LaTeX: a complete article that pdfLaTeX, XeLaTeX and LuaLaTeX take with the packages of a plain TeX +// installation (iftex, fontenc / inputenc or fontspec, amsmath, amssymb, graphicx, longtable, array, multirow, ulem, +// hyperref; only those the document needs). Text outside the Latin scripts needs XeLaTeX or LuaLaTeX and a font that +// has it; CJK text loads xeCJK or luatexja there. +// +// \section … \subparagraph headings (unnumbered, as the source numbers its own), paragraphs with \\ at line breaks, +// quote environments for quotes, itemize / enumerate nested up to LaTeX's four levels (numbers written as the items' +// labels), verbatim code (\texttt lines where verbatim can't go: in table cells, or code holding \end{verbatim}), +// longtable tables (tabular inside a cell) with p columns, \multicolumn and \multirow for merged cells and \cline +// rules that leave merged cells open, header rows repeated on every page, \href links, \includegraphics pictures +// (files in "_files" next to the result, no wider than the text), \footnote notes, rules, and equations as +// $…$ / \[…\] (their TeX source, or TeX written from the equation tree). Every character LaTeX treats specially is +// escaped: \ { } $ & # _ % ~ ^, the brackets after \\ and \item, < > | " ` and the ligatures -- '' ?` !`. + +using System.Globalization; +using System.Text; + +namespace Filee.Engines.Hwp.Hwpx.Markup; + +/// Writes the document model as LaTeX. +internal sealed class LatexWriter(Func pictures) : MarkupTextWriter(pictures) +{ + /// LaTeX nests lists four levels deep. + private const int MaxListLevel = 3; + + /// Environments that are a display equation by themselves (a source written $$\begin{align}…$$). + private static readonly string[] DisplayEnvironments = ["align", "equation", "gather", "multline", "eqnarray", "flalign", "alignat"]; + + private bool _pictures; + private bool _tables; + private bool _lines; + private bool _cjk; + + /// Table cells around the text being written: tables there are tabular, code is \texttt. + private int _cell; + + public static string Write(HDocument document, Func pictures) => new LatexWriter(pictures).Write(document); + + protected override void Start(StringBuilder sb, string? title) + { + sb.Append("\\begin{document}\n\n"); + if (title is null) + return; + sb.Append("\\title{").Append(Escape(title.Replace('\n', ' '))).Append("}\n"); + var authors = Source?.Authors.Where(a => !string.IsNullOrWhiteSpace(a)).Select(a => Escape(a.Trim())).ToList() ?? []; + sb.Append("\\author{").AppendJoin(" \\and ", authors).Append("}\n"); + sb.Append("\\date{").Append(Escape(Source?.Date?.Trim() ?? "")).Append("}\n\\maketitle\n\n"); + } + + protected override void End(StringBuilder sb) + { + sb.Append("\\end{document}\n"); + // The preamble loads what the body turned out to use. + sb.Insert(0, Preamble()); + } + + private string Preamble() + { + var sb = new StringBuilder(); + sb.Append("\\documentclass{article}\n"); + sb.Append("\\usepackage{iftex}\n"); + sb.Append("\\ifPDFTeX\n \\usepackage[T1]{fontenc}\n \\usepackage[utf8]{inputenc}\n\\else\n \\usepackage{fontspec}\n\\fi\n"); + if (_cjk) + sb.Append("\\ifXeTeX\n \\usepackage{xeCJK}\n\\fi\n\\ifLuaTeX\n \\usepackage{luatexja}\n\\fi\n"); + sb.Append("\\usepackage{amsmath,amssymb}\n"); + if (_pictures) + { + // Pictures at their own size, but no wider than the text. + sb.Append("\\usepackage{graphicx}\n"); + sb.Append("\\makeatletter\n\\def\\maxwidth{\\ifdim\\Gin@nat@width>\\linewidth\\linewidth\\else\\Gin@nat@width\\fi}\n\\makeatother\n"); + sb.Append("\\setkeys{Gin}{width=\\maxwidth,keepaspectratio}\n"); + } + if (_tables) + sb.Append("\\usepackage{longtable,array,multirow}\n"); + if (_lines) + sb.Append("\\usepackage[normalem]{ulem}\n"); + sb.Append("\\usepackage{hyperref}\n"); + sb.Append("\\setcounter{secnumdepth}{-2}\n"); + sb.Append("\\setlength{\\parindent}{0pt}\n\\setlength{\\parskip}{6pt plus 2pt minus 1pt}\n\n"); + return sb.ToString(); + } + + protected override void Heading(StringBuilder sb, int level, string text) + { + if (text.Length == 0) + return; + if (_cell > 0) + { + sb.Append("\\textbf{").Append(text).Append("}\n\n"); + return; + } + var command = level switch + { + 1 => "section", + 2 => "subsection", + 3 => "subsubsection", + 4 => "paragraph", + _ => "subparagraph", + }; + sb.Append('\\').Append(command).Append('{').Append(text).Append("}\n\n"); + } + + protected override void Paragraph(StringBuilder sb, string text, int quote) => + Quoted(sb, quote, () => sb.Append(Lines(text)).Append("\n\n")); + + private static void Quoted(StringBuilder sb, int quote, Action body) + { + for (var q = 0; q < quote; q++) + sb.Append("\\begin{quote}\n"); + body(); + for (var q = 0; q < quote; q++) + sb.Append("\\end{quote}\n"); + if (quote > 0) + sb.Append('\n'); + } + + /// A paragraph's lines joined by \\ (an empty line holds an empty box, as \\ can't end nothing). + private static string Lines(string text) => + string.Join("\\\\\n", text.Trim('\n').Split('\n').Select(line => line.Trim().Length == 0 ? "\\mbox{}" : line)); + + protected override void List(StringBuilder sb, IReadOnlyList items, int quote) + { + // The environment open at each level, from the outside in. + var open = new List(); + foreach (var item in items) + { + if (item.Continuation && open.Count > 0) + { + sb.Append('\n').Append(Lines(item.Text)).Append('\n'); + continue; + } + var level = Math.Min(item.Level, MaxListLevel); + var environment = item.Ordered ? "enumerate" : "itemize"; + while (open.Count > level + 1 || (open.Count == level + 1 && open[^1] != environment)) + { + sb.Append("\\end{").Append(open[^1]).Append("}\n"); + open.RemoveAt(open.Count - 1); + } + while (open.Count < level + 1) + { + var opening = open.Count == level ? environment : "itemize"; + sb.Append("\\begin{").Append(opening).Append("}\n"); + open.Add(opening); + // A level skipped over still needs an item to hold the next one. + if (open.Count <= level) + sb.Append("\\item[]\n"); + } + sb.Append("\\item"); + if (item.Continuation) + sb.Append("[]"); + else if (item.Ordered) + sb.Append(CultureInfo.InvariantCulture, $"[{item.Number}.]"); + sb.Append(' ').Append(Lines(item.Text)).Append('\n'); + } + for (var i = open.Count - 1; i >= 0; i--) + sb.Append("\\end{").Append(open[i]).Append("}\n"); + sb.Append('\n'); + } + + protected override void Code(StringBuilder sb, IReadOnlyList lines, int quote) + { + var expanded = lines.Select(l => l.Replace("\t", " ", StringComparison.Ordinal)).ToList(); + Quoted(sb, quote, () => + { + if (_cell == 0 && !expanded.Any(l => l.Contains("\\end{verbatim}", StringComparison.Ordinal))) + { + sb.Append("\\begin{verbatim}\n"); + foreach (var line in expanded) + sb.Append(line).Append('\n'); + sb.Append("\\end{verbatim}\n\n"); + return; + } + // Spaces kept, each line its own. + sb.Append("{\\ttfamily ").AppendJoin("\\\\\n", expanded.Select(l => l.Length == 0 ? "\\mbox{}" : Escape(l).Replace(" ", "\\ ", StringComparison.Ordinal))) + .Append("\\par}\n\n"); + }); + } + + protected override void Table(StringBuilder sb, MarkupTable table) + { + _tables = true; + var columns = table.Columns; + var owners = MarkupTableOwners.Owners(table); + var nested = _cell > 0; + var environment = nested ? "tabular" : "longtable"; + // Columns share the line; a merged cell is as wide as its columns with the space between them. + string Width(int span) => + $"\\dimexpr {((double)span / columns).ToString("0.####", CultureInfo.InvariantCulture)}\\linewidth-2\\tabcolsep-\\arrayrulewidth\\relax"; + + sb.Append("\\begin{").Append(environment).Append("}{|"); + for (var c = 0; c < columns; c++) + sb.Append("p{").Append(Width(1)).Append("}|"); + sb.Append("}\n\\hline\n"); + var header = 0; + while (header < table.Rows.Count && table.HeaderRows[header]) + header++; + + for (var r = 0; r < table.Rows.Count; r++) + { + var entries = new List(); + for (var c = 0; c < columns;) + { + var (ownerRow, ownerColumn) = owners[r, c]; + var owner = ownerRow < 0 ? null : table.Rows[ownerRow][ownerColumn]; + var span = owner?.ColSpan ?? 1; + var content = owner is not null && ownerRow == r ? Content(owner) : ""; + if (owner is not null && ownerRow == r && owner.RowSpan > 1 && owner.Simple) + content = $"\\multirow{{{owner.RowSpan}}}{{=}}{{{content}}}"; + if (span > 1) + content = $"\\multicolumn{{{span}}}{{{(c == 0 ? "|" : "")}p{{{Width(span)}}}|}}{{{content}}}"; + entries.Add(content); + c += span; + } + sb.AppendJoin(" & ", entries).Append(" \\\\\n"); + sb.Append(Rules(table, owners, r)).Append('\n'); + if (!nested && r + 1 == header && header < table.Rows.Count) + sb.Append("\\endhead\n"); + } + sb.Append("\\end{").Append(environment).Append("}\n"); + if (!nested) + sb.Append('\n'); + } + + /// \hline under a row, or \cline over the columns whose cell ends there (merged cells stay open). + private static string Rules(MarkupTable table, (int Row, int Column)[,] owners, int row) + { + bool Ends(int c) + { + var (ownerRow, ownerColumn) = owners[row, c]; + return ownerRow < 0 || ownerRow + table.Rows[ownerRow][ownerColumn]!.RowSpan - 1 <= row; + } + + var ends = Enumerable.Range(0, table.Columns).Select(Ends).ToArray(); + if (ends.All(e => e)) + return "\\hline"; + var sb = new StringBuilder(); + for (var c = 0; c < ends.Length; c++) + { + if (!ends[c]) + continue; + var start = c; + while (c + 1 < ends.Length && ends[c + 1]) + c++; + sb.Append(CultureInfo.InvariantCulture, $"\\cline{{{start + 1}-{c + 1}}}"); + } + return sb.ToString(); + } + + private string Content(MarkupCell cell) + { + if (cell.Simple) + return string.Join("\\newline ", cell.Text.Split('\n').Where(l => l.Length > 0)); + var blocks = new StringBuilder(); + _cell++; + try + { + Blocks(cell.Source.Blocks, blocks); + } + finally + { + _cell--; + } + return blocks.ToString().Trim('\n'); + } + + protected override void Rule(StringBuilder sb) => sb.Append("\\noindent\\rule{\\linewidth}{0.4pt}\n\n"); + + protected override string Run(string text, HCharFormat format) + { + if (format.Shade == HtmlReader.CodeShade) + return Around(text, core => $"\\texttt{{{Escape(core)}}}"); + return Around(Escape(text), core => + { + if (format.Superscript is true) + core = $"\\textsuperscript{{{core}}}"; + else if (format.Subscript is true) + core = $"\\textsubscript{{{core}}}"; + if (format.Underline is true || format.Strike is true) + _lines = true; + if (format.Underline is true) + core = $"\\protect\\uline{{{core}}}"; + if (format.Strike is true) + core = $"\\protect\\sout{{{core}}}"; + if (format.Italic is true) + core = $"\\emph{{{core}}}"; + if (format.Bold is true) + core = $"\\textbf{{{core}}}"; + return core; + }); + } + + protected override string Link(string target, string content) => $"\\href{{{Url(target)}}}{{{content}}}"; + + protected override string Picture(string link) + { + _pictures = true; + // The link is a URL path; LaTeX wants the file's name. + return $"\\includegraphics{{{Uri.UnescapeDataString(link)}}}"; + } + + protected override string LineBreak() => "\n"; + + protected override string NoteReference(int number) => $"\\protect\\footnote{{{Notes[number - 1]}}}"; + + protected override string? Equation(HEquation equation) + { + if (Tex(equation) is not { } tex) + return null; + _cjk |= tex.Any(IsCjk); + return equation.Display ? $"\\[ {tex} \\]" : $"${tex}$"; + } + + protected override bool DisplayEquation(StringBuilder sb, HEquation equation, int quote) + { + if (equation.Tex is { } source && IsDisplayEnvironment(source) && _cell == 0) + { + Quoted(sb, quote, () => sb.Append(source.Trim()).Append("\n\n")); + return true; + } + if (Tex(equation) is not { } tex) + return false; + _cjk |= tex.Any(IsCjk); + Quoted(sb, quote, () => sb.Append("\\[\n").Append(tex).Append("\n\\]\n\n")); + return true; + } + + /// + /// An equation's TeX: its source, unless that is a display environment of its own (which $…$ can't hold), else + /// written from its tree; null when neither can be had. + /// + private static string? Tex(HEquation equation) => + equation.Tex is { } source && !IsDisplayEnvironment(source) && source.Trim().Length > 0 ? source.Trim() : TexMathWriter.Write(equation.Math); + + private static bool IsDisplayEnvironment(string tex) + { + var trimmed = tex.TrimStart(); + return DisplayEnvironments.Any(e => trimmed.StartsWith("\\begin{" + e, StringComparison.Ordinal)); + } + + // ───────────────────────── Escaping ───────────────────────── + + /// Text with every character LaTeX treats specially written so it prints as itself. + private string Escape(string text) + { + var sb = new StringBuilder(text.Length + 16); + for (var i = 0; i < text.Length; i++) + { + var ch = text[i]; + var next = i + 1 < text.Length ? text[i + 1] : '\0'; + if (IsCjk(ch)) + _cjk = true; + switch (ch) + { + case '\\': + sb.Append("\\textbackslash{}"); + break; + case '{' or '}' or '$' or '&' or '#' or '_' or '%': + sb.Append('\\').Append(ch); + break; + case '~': + sb.Append("\\textasciitilde{}"); + break; + case '^': + sb.Append("\\textasciicircum{}"); + break; + case '<': + sb.Append("\\textless{}"); + break; + case '>': + sb.Append("\\textgreater{}"); + break; + case '|': + sb.Append("\\textbar{}"); + break; + case '"': + sb.Append("\\textquotedbl{}"); + break; + case '`': + sb.Append("\\textasciigrave{}"); + break; + case '[' or ']': + // In braces: a bracket after \\ or \item would be read as an option. + sb.Append('{').Append(ch).Append('}'); + break; + case '-' when next == '-': + sb.Append("-{}"); + break; + case '\'' when next == '\'': + sb.Append("'{}"); + break; + case '\u00A0': + sb.Append('~'); + break; + case '\u00AD': + sb.Append("\\-"); + break; + case '\t': + sb.Append(' '); + break; + case '\n': + sb.Append('\n'); + break; + case < ' ' or '\u200B' or '\uFEFF': + break; + default: + sb.Append(ch); + break; + } + } + return sb.ToString(); + } + + /// A URL for \href: the characters an argument can't hold percent-encoded or escaped. + private static string Url(string target) + { + var sb = new StringBuilder(target.Length + 8); + foreach (var ch in target) + { + switch (ch) + { + case '%' or '#': + sb.Append('\\').Append(ch); + break; + case '\\' or '{' or '}' or '~' or '^' or ' ' or '"' or '<' or '>' or '|' or '`': + sb.Append(CultureInfo.InvariantCulture, $"%{(int)ch:X2}"); + break; + case < ' ': + break; + default: + sb.Append(ch); + break; + } + } + return sb.ToString(); + } + + /// Hangul, kana, CJK ideographs and their punctuation: text that needs xeCJK or luatexja. + internal static bool IsCjk(char ch) => + ch is >= '\u1100' and <= '\u11FF' or >= '\u2E80' and <= '\u9FFF' or >= '\uA960' and <= '\uA97F' or >= '\uAC00' and <= '\uD7FF' + or >= '\uF900' and <= '\uFAFF' or >= '\uFE30' and <= '\uFE4F' or >= '\uFF00' and <= '\uFFEF'; +} diff --git a/src/Filee.Engines/Hwp/Hwpx/Markup/ManWriter.cs b/src/Filee.Engines/Hwp/Hwpx/Markup/ManWriter.cs new file mode 100644 index 000000000..66d1ef5f7 --- /dev/null +++ b/src/Filee.Engines/Hwp/Hwpx/Markup/ManWriter.cs @@ -0,0 +1,237 @@ +// HDocument → a man page (the man macros, as groff and mandoc read them): .TH with the title, .SH / .SS headings +// (deeper ones as bold paragraphs), .PP paragraphs with .br at line breaks, .RS / .RE indents for quotes and nested +// lists, .IP items with a bullet or their number, .EX / .EE code, tbl tables (.TS … .TE, boxed, header rows bold, s and +// ^ for merged cells, every cell a T{ … T} text block), \fB \fI \f(CR fonts, links as their text and , and +// footnotes numbered [1] in the text and listed under NOTES at the end. A man page has no pictures: they are left out. +// Equations are their linear text ("(a)/(b)"). +// +// Escaping: the file is ASCII. Backslashes are \e, minus signs \-, quotes, apostrophes, backticks, carets and tildes +// their named characters (\(dq \(aq \(ga \(ha \(ti), other characters \[uXXXX]; a line that starts with "." or "'" +// (a request), a space (a break) or "T}" (the end of a table cell) gets \& first. + +using System.Globalization; +using System.Text; + +namespace Filee.Engines.Hwp.Hwpx.Markup; + +/// Writes the document model as a man page. +internal sealed class ManWriter : MarkupTextWriter +{ + /// Pictures are left out: a man page can't show them. + public ManWriter() + : base(_ => null) + { + } + + protected override void Start(StringBuilder sb, string? title) + { + var name = title is null ? "" : Escape(title.Replace('\n', ' ')); + var date = Source?.Date is { } d ? Escape(d.Trim()) : ""; + sb.Append(".TH \"").Append(name).Append("\" \"1\" \"").Append(date).Append("\" \"\" \"\"\n"); + } + + protected override void End(StringBuilder sb) + { + if (Notes.Count == 0) + return; + sb.Append(".SH NOTES\n"); + for (var i = 0; i < Notes.Count; i++) + sb.Append(CultureInfo.InvariantCulture, $".IP \"[{i + 1}]\" 5\n").Append(Lines(Notes[i])).Append('\n'); + } + + protected override void Heading(StringBuilder sb, int level, string text) + { + if (text.Length == 0) + return; + if (level <= 2) + sb.Append(level == 1 ? ".SH \"" : ".SS \"").Append(text).Append("\"\n"); + else + sb.Append(".PP\n").Append(Lines("\\fB" + text + "\\fP")).Append('\n'); + } + + protected override void Paragraph(StringBuilder sb, string text, int quote) + { + for (var q = 0; q < quote; q++) + sb.Append(".RS 4\n"); + sb.Append(".PP\n").Append(Lines(text)).Append('\n'); + for (var q = 0; q < quote; q++) + sb.Append(".RE\n"); + } + + /// Text lines, .br between the lines of a break, each guarded against reading as a request. + private static string Lines(string text) => + string.Join("\n.br\n", text.Trim('\n').Split('\n').Select(LineStart)); + + protected override void List(StringBuilder sb, IReadOnlyList items, int quote) + { + var depth = 0; + foreach (var item in items) + { + for (; depth < item.Level; depth++) + sb.Append(".RS 4\n"); + for (; depth > item.Level; depth--) + sb.Append(".RE\n"); + var tag = item.Continuation ? "" : item.Ordered ? item.Number.ToString(CultureInfo.InvariantCulture) + "." : "\\(bu"; + sb.Append(".IP \"").Append(tag).Append("\" 4\n").Append(Lines(item.Text)).Append('\n'); + } + for (; depth > 0; depth--) + sb.Append(".RE\n"); + } + + protected override void Code(StringBuilder sb, IReadOnlyList lines, int quote) + { + for (var q = 0; q < quote; q++) + sb.Append(".RS 4\n"); + sb.Append(".PP\n.EX\n"); + foreach (var line in lines) + sb.Append(LineStart(Escape(line))).Append('\n'); + sb.Append(".EE\n"); + for (var q = 0; q < quote; q++) + sb.Append(".RE\n"); + } + + protected override void Table(StringBuilder sb, MarkupTable table) + { + var owners = MarkupTableOwners.Owners(table); + sb.Append(".PP\n.TS\nallbox tab(@);\n"); + // A format line per row: l for a cell (lb in a header row), s where a cell spans from the left, ^ from above. + for (var r = 0; r < table.Rows.Count; r++) + { + var keys = new List(table.Columns); + for (var c = 0; c < table.Columns; c++) + { + var (ownerRow, ownerColumn) = owners[r, c]; + keys.Add(ownerRow < r && ownerRow >= 0 && ownerColumn == c ? "^" + : ownerRow >= 0 && ownerColumn < c ? "s" + : table.HeaderRows[r] ? "lb" : "l"); + } + sb.AppendJoin(' ', keys).Append(r + 1 == table.Rows.Count ? ".\n" : "\n"); + } + for (var r = 0; r < table.Rows.Count; r++) + { + var entries = new List(table.Columns); + for (var c = 0; c < table.Columns; c++) + { + var (ownerRow, ownerColumn) = owners[r, c]; + entries.Add((ownerRow, ownerColumn) == (r, c) && table.Rows[r][c] is { } cell + ? "T{\n" + Lines(CellText(cell).Replace("@", "\\(at", StringComparison.Ordinal)) + "\nT}" + : ""); + } + // Entries after the last cell of the row are left blank. + while (entries.Count > 0 && entries[^1].Length == 0) + entries.RemoveAt(entries.Count - 1); + sb.AppendJoin('@', entries).Append('\n'); + } + sb.Append(".TE\n"); + } + + /// A cell's text: its paragraphs, and those of the tables in it, as lines. + private string CellText(MarkupCell cell) + { + if (cell.Simple) + return cell.Text; + var lines = new List(); + var pending = new Stack(Enumerable.Reverse(cell.Source.Blocks)); + while (pending.TryPop(out var block)) + { + switch (block) + { + case HParagraph paragraph when Inlines(paragraph.Inlines).Trim() is { Length: > 0 } text: + lines.Add(text.Replace("\n", " ", StringComparison.Ordinal)); + break; + case HTable inner: + foreach (var nested in Enumerable.Reverse(inner.Rows.SelectMany(r => r.Cells).SelectMany(c => c.Blocks).ToList())) + pending.Push(nested); + break; + } + } + return string.Join('\n', lines); + } + + protected override void Rule(StringBuilder sb) => sb.Append(".PP\n.ce 1\n* * *\n"); + + protected override string Run(string text, HCharFormat format) + { + if (format.Shade == HtmlReader.CodeShade) + return Around(Escape(text), core => $"\\f(CR{core}\\fP"); + return Around(Escape(text), core => (format.Bold is true, format.Italic is true || format.Underline is true) switch + { + (true, true) => $"\\f(BI{core}\\fP", + (true, false) => $"\\fB{core}\\fP", + (false, true) => $"\\fI{core}\\fP", + _ => core, + }); + } + + protected override string Link(string target, string content) + { + var url = Escape(target); + return content.Trim() == url ? content : $"{content} <{url}>"; + } + + protected override string Picture(string link) => ""; + + protected override string LineBreak() => "\n"; + + protected override string NoteReference(int number) => $"[{number}]"; + + protected override string? Equation(HEquation equation) => + EquationScript.Deeper(equation.Math, TexMathWriter.MaxDepth) ? null : Escape(EquationScript.Linear(equation.Math)); + + // ───────────────────────── Escaping ───────────────────────── + + /// Text as roff prints it: ASCII, with the characters roff reads specially escaped. + private static string Escape(string text) + { + var sb = new StringBuilder(text.Length + 16); + foreach (var rune in text.EnumerateRunes()) + { + switch (rune.Value) + { + case '\\': + sb.Append("\\e"); + break; + case '-': + sb.Append("\\-"); + break; + case '"': + sb.Append("\\(dq"); + break; + case '\'': + sb.Append("\\(aq"); + break; + case '`': + sb.Append("\\(ga"); + break; + case '^': + sb.Append("\\(ha"); + break; + case '~': + sb.Append("\\(ti"); + break; + case '\n': + sb.Append('\n'); + break; + case '\t': + sb.Append(' '); + break; + case 0xA0: + sb.Append("\\ "); + break; + case < ' ' or 0x7F or 0xAD or 0xFEFF: + break; + case < 0x80: + sb.Append((char)rune.Value); + break; + default: + sb.Append(CultureInfo.InvariantCulture, $"\\[u{rune.Value:X4}]"); + break; + } + } + return sb.ToString(); + } + + /// A text line that would read as a request, a break or the end of a table cell gets \& first. + private static string LineStart(string line) => + line.Length > 0 && (line[0] is '.' or '\'' or ' ' || line.StartsWith("T}", StringComparison.Ordinal)) ? "\\&" + line : line; +} diff --git a/src/Filee.Engines/Hwp/Hwpx/Markup/MarkupTableOwners.cs b/src/Filee.Engines/Hwp/Hwpx/Markup/MarkupTableOwners.cs new file mode 100644 index 000000000..4c558b294 --- /dev/null +++ b/src/Filee.Engines/Hwp/Hwpx/Markup/MarkupTableOwners.cs @@ -0,0 +1,29 @@ +// Which cell holds each position of a MarkupTable's grid: the writers whose tables say what a merged cell covers +// (LaTeX's \multicolumn and rules, AsciiDoc's and Typst's cell counts, tbl's span columns) tell a position covered by a +// span from one no cell has. The grid is counted against the budget before it is made (MarkupTextWriter.Grid). + +namespace Filee.Engines.Hwp.Hwpx.Markup; + +internal static class MarkupTableOwners +{ + /// The row and column of the cell holding each grid position, or (-1, -1) where none does. + public static (int Row, int Column)[,] Owners(MarkupTable table) + { + var owners = new (int Row, int Column)[table.Rows.Count, table.Columns]; + for (var r = 0; r < table.Rows.Count; r++) + for (var c = 0; c < table.Columns; c++) + owners[r, c] = (-1, -1); + for (var r = 0; r < table.Rows.Count; r++) + { + for (var c = 0; c < table.Columns; c++) + { + if (table.Rows[r][c] is not { } cell) + continue; + for (var rr = r; rr < Math.Min(table.Rows.Count, r + cell.RowSpan); rr++) + for (var cc = c; cc < Math.Min(table.Columns, c + cell.ColSpan); cc++) + owners[rr, cc] = (r, c); + } + } + return owners; + } +} diff --git a/src/Filee.Engines/Hwp/Hwpx/Markup/MarkupTextWriter.cs b/src/Filee.Engines/Hwp/Hwpx/Markup/MarkupTextWriter.cs index fe1603d8c..f480c4819 100644 --- a/src/Filee.Engines/Hwp/Hwpx/Markup/MarkupTextWriter.cs +++ b/src/Filee.Engines/Hwp/Hwpx/Markup/MarkupTextWriter.cs @@ -49,11 +49,18 @@ internal abstract class MarkupTextWriter /// The caps of this document's tables and text. protected MarkupWriteBudget Budget => _budget ??= new(Limits); + /// The document being written (for its metadata), set by . + protected HDocument? Source { get; private set; } + + /// Stops the walk between blocks when the conversion is cancelled. + public CancellationToken Cancellation { get; init; } + /// The text of the document. public string Write(HDocument document) { // The walk recurses per table, link, note and text box level: deeper content is opened up into paragraphs. Nesting.Limit(document); + Source = document; var sb = new StringBuilder(); Start(sb, string.IsNullOrWhiteSpace(document.Title) ? null : document.Title.Trim()); foreach (var section in document.Sections) @@ -97,6 +104,15 @@ public string Write(HDocument document) /// The reference to footnote (its text is Notes[number - 1]). protected abstract string NoteReference(int number); + /// An equation among the text, or null to write its text (TeX source or linear form) as a run. + protected virtual string? Equation(HEquation equation) => null; + + /// + /// A paragraph that is a display equation alone (); false leaves it to + /// . + /// + protected virtual bool DisplayEquation(StringBuilder sb, HEquation equation, int quote) => false; + // ───────────────────────── The walk ───────────────────────── /// Quote level of a paragraph: the readers indent quotes by 2000 HWPUNIT per level. @@ -107,6 +123,7 @@ protected void Blocks(List blocks, StringBuilder sb) for (var i = 0; i < blocks.Count;) { Budget.Characters(sb.Length + _noteCharacters); + Cancellation.ThrowIfCancellationRequested(); var block = blocks[i]; if (MarkupBlocks.IsCode(block)) { @@ -145,6 +162,8 @@ protected void Blocks(List blocks, StringBuilder sb) case HParagraph paragraph when paragraph.Inlines.Any(x => x is HShape { Kind: HShapeKind.Line }) && HDocumentWalker.Text(paragraph.Inlines).Length == 0: Rule(sb); break; + case HParagraph paragraph when LoneEquation(paragraph) is { } equation && DisplayEquation(sb, equation, Quote(paragraph)): + break; case HParagraph paragraph: { var text = Inlines(paragraph.Inlines); @@ -160,6 +179,10 @@ protected void Blocks(List blocks, StringBuilder sb) Budget.Characters(sb.Length + _noteCharacters); } + /// The equation a paragraph holds alone (bookmarks aside) when it is a display equation, else null. + private static HEquation? LoneEquation(HParagraph paragraph) => + paragraph.Inlines.Where(i => i is not HBookmark).ToList() is [HEquation equation] && HEquation.IsDisplay(equation, paragraph) ? equation : null; + /// /// The table as a grid; a cell's paragraphs are joined by line breaks. Spans are placed arithmetically /// () and the grid is counted before it is made. @@ -209,6 +232,12 @@ void Flush() foreach (var inline in inlines) { + if (inline is HEquation equation && Equation(equation) is { } math) + { + Flush(); + sb.Append(math); + continue; + } if (inline is HText t) { var plain = WithoutLayout(t.Format); diff --git a/src/Filee.Engines/Hwp/Hwpx/Markup/TypstWriter.cs b/src/Filee.Engines/Hwp/Hwpx/Markup/TypstWriter.cs new file mode 100644 index 000000000..9ef741543 --- /dev/null +++ b/src/Filee.Engines/Hwp/Hwpx/Markup/TypstWriter.cs @@ -0,0 +1,280 @@ +// HDocument → Typst markup (Typst 0.11 or later, for table.cell and table.header): the title as the document's +// metadata and a centred heading line, = headings, paragraphs with \ at line breaks, #quote(block: true) for quotes, +// - and "N." lists nested by indentation (an item's further paragraphs indented under it), ``` raw blocks, #table with +// table.header rows and table.cell(colspan, rowspan) for merged cells, #line rules, #strong #emph #underline #strike +// #super #sub #highlight and `raw` text, #link, #image pictures (files in "_files" next to the result), +// #footnote notes, and equations in Typst's own math ($…$ inline, $ … $ on their own), written from the equation tree. +// +// Escaping: every markup character (\ # $ * _ ` < > @ [ ] ~ ' ") gets a backslash, as do // and /* (comments), -- and +// -? (dashes, soft hyphens), ... and :// (an ellipsis, a URL), a "." or "(" that starts a run (it would continue the +// #function before it), and a line that would start a heading, list or term (= - + / "1."). + +using System.Globalization; +using System.Text; + +namespace Filee.Engines.Hwp.Hwpx.Markup; + +/// Writes the document model as Typst markup. +internal sealed class TypstWriter(Func pictures) : MarkupTextWriter(pictures) +{ + public static string Write(HDocument document, Func pictures) => new TypstWriter(pictures).Write(document); + + protected override void Start(StringBuilder sb, string? title) + { + var authors = Source?.Authors.Where(a => !string.IsNullOrWhiteSpace(a)).Select(a => Quoted(a.Trim())).ToList() ?? []; + if (title is null && authors.Count == 0) + return; + sb.Append("#set document("); + if (title is not null) + sb.Append("title: ").Append(Quoted(title.Replace('\n', ' '))); + if (authors.Count > 0) + sb.Append(title is null ? "" : ", ").Append("author: (").AppendJoin(", ", authors).Append(authors.Count == 1 ? ",)" : ")"); + sb.Append(")\n\n"); + if (title is not null) + sb.Append("#align(center, text(size: 1.7em, weight: \"bold\")[").Append(Escape(title.Replace('\n', ' '))).Append("])\n\n"); + } + + protected override void End(StringBuilder sb) + { + } + + protected override void Heading(StringBuilder sb, int level, string text) + { + if (text.Length > 0) + sb.Append('=', level).Append(' ').Append(text).Append("\n\n"); + } + + protected override void Paragraph(StringBuilder sb, string text, int quote) => + Quoted(sb, quote, () => sb.Append(Lines(text, "")).Append("\n\n")); + + private static void Quoted(StringBuilder sb, int quote, Action body) + { + for (var q = 0; q < quote; q++) + sb.Append("#quote(block: true)[\n"); + body(); + for (var q = 0; q < quote; q++) + sb.Append("]\n"); + if (quote > 0) + sb.Append('\n'); + } + + /// A paragraph's lines, \ ending each that breaks, each guarded against starting a block. + private static string Lines(string text, string indent) => + string.Join(" \\\n" + indent, text.Trim('\n').Split('\n').Select(line => LineStart(line.Trim()))); + + protected override void List(StringBuilder sb, IReadOnlyList items, int quote) + { + foreach (var item in items) + { + var indent = new string(' ', item.Level * 2); + if (item.Continuation) + { + // A further paragraph of the item: after a blank line, indented under its text. + var under = indent + " "; + sb.Append('\n').Append(under).Append(Lines(item.Text, under)).Append('\n'); + continue; + } + var marker = item.Ordered ? item.Number.ToString(CultureInfo.InvariantCulture) + ". " : "- "; + sb.Append(indent).Append(marker).Append(Lines(item.Text, indent + new string(' ', marker.Length))).Append('\n'); + } + sb.Append('\n'); + } + + protected override void Code(StringBuilder sb, IReadOnlyList lines, int quote) + { + var longest = lines.Select(l => LongestRun(l, '`')).DefaultIfEmpty(0).Max(); + var fence = new string('`', Math.Max(3, longest + 1)); + Quoted(sb, quote, () => + { + sb.Append(fence).Append('\n'); + foreach (var line in lines) + sb.Append(line).Append('\n'); + sb.Append(fence).Append("\n\n"); + }); + } + + private static int LongestRun(string text, char ch) + { + int longest = 0, run = 0; + foreach (var c in text) + { + run = c == ch ? run + 1 : 0; + longest = Math.Max(longest, run); + } + return longest; + } + + protected override void Table(StringBuilder sb, MarkupTable table) + { + var owners = MarkupTableOwners.Owners(table); + sb.Append(CultureInfo.InvariantCulture, $"#table(\n columns: {table.Columns},\n"); + var header = 0; + while (header < table.Rows.Count && table.HeaderRows[header]) + header++; + if (header > 0) + sb.Append(" table.header(\n"); + for (var r = 0; r < table.Rows.Count; r++) + { + var indent = r < header ? " " : " "; + for (var c = 0; c < table.Columns; c++) + { + var (ownerRow, ownerColumn) = owners[r, c]; + if (ownerRow >= 0 && (ownerRow, ownerColumn) != (r, c)) + continue; // covered by a merged cell + sb.Append(indent); + if (table.Rows[r][c] is not { } cell) + { + sb.Append("[],\n"); + continue; + } + if (cell.ColSpan > 1 || cell.RowSpan > 1) + { + sb.Append("table.cell("); + if (cell.ColSpan > 1) + sb.Append(CultureInfo.InvariantCulture, $"colspan: {cell.ColSpan}"); + if (cell.RowSpan > 1) + sb.Append(cell.ColSpan > 1 ? ", " : "").Append(CultureInfo.InvariantCulture, $"rowspan: {cell.RowSpan}"); + sb.Append(')'); + } + sb.Append('[').Append(Content(cell)).Append("],\n"); + } + if (r + 1 == header) + sb.Append(" ),\n"); + } + sb.Append(")\n\n"); + } + + private string Content(MarkupCell cell) + { + if (cell.Simple) + return Lines(cell.Text, ""); + var blocks = new StringBuilder(); + Blocks(cell.Source.Blocks, blocks); + return "\n" + blocks.ToString().Trim('\n') + "\n"; + } + + protected override void Rule(StringBuilder sb) => sb.Append("#line(length: 100%)\n\n"); + + protected override string Run(string text, HCharFormat format) + { + if (format.Shade == HtmlReader.CodeShade) + return Around(text, core => core.Contains('`', StringComparison.Ordinal) ? $"#raw({Quoted(core)})" : $"`{core}`"); + return Around(Escape(text), core => + { + if (format.Superscript is true) + core = $"#super[{core}]"; + else if (format.Subscript is true) + core = $"#sub[{core}]"; + if (format.Underline is true) + core = $"#underline[{core}]"; + if (format.Strike is true) + core = $"#strike[{core}]"; + if (format.Shade == HtmlReader.MarkShade) + core = $"#highlight[{core}]"; + if (format.Italic is true) + core = $"#emph[{core}]"; + if (format.Bold is true) + core = $"#strong[{core}]"; + return core; + }); + } + + protected override string Link(string target, string content) => $"#link({Quoted(target)})[{content}]"; + + // The link is a URL path; Typst wants the file's name. + protected override string Picture(string link) => $"#image({Quoted(Uri.UnescapeDataString(link))})"; + + protected override string LineBreak() => "\n"; + + protected override string NoteReference(int number) => $"#footnote[{Notes[number - 1]}]"; + + protected override string? Equation(HEquation equation) + { + if (TypstMathWriter.Write(equation.Math) is not { Length: > 0 } math) + return null; + return equation.Display ? $"$ {math} $" : $"${math}$"; + } + + protected override bool DisplayEquation(StringBuilder sb, HEquation equation, int quote) + { + if (TypstMathWriter.Write(equation.Math) is not { Length: > 0 } math) + return false; + Quoted(sb, quote, () => sb.Append("$ ").Append(math).Append(" $\n\n")); + return true; + } + + // ───────────────────────── Escaping ───────────────────────── + + /// A Typst string literal. + private static string Quoted(string text) + { + var sb = new StringBuilder(text.Length + 2).Append('"'); + foreach (var ch in text) + { + sb.Append(ch switch + { + '\\' => "\\\\", + '"' => "\\\"", + '\n' => "\\n", + '\r' => "\\r", + '\t' => "\\t", + _ => ch.ToString(), + }); + } + return sb.Append('"').ToString(); + } + + /// Text with every character Typst's markup gives a meaning escaped. + private static string Escape(string text) + { + var sb = new StringBuilder(text.Length + 16); + for (var i = 0; i < text.Length; i++) + { + var ch = text[i]; + var next = i + 1 < text.Length ? text[i + 1] : '\0'; + var escape = ch switch + { + '\\' or '#' or '$' or '*' or '_' or '`' or '<' or '>' or '@' or '[' or ']' or '~' or '\'' or '"' => true, + '/' => next is '/' or '*', + '-' => next is '-' or '?', + '.' => i == 0 || (next == '.' && i + 2 < text.Length && text[i + 2] == '.'), + '(' => i == 0, + ':' => next == '/' && i + 2 < text.Length && text[i + 2] == '/', + _ => false, + }; + if (escape) + { + sb.Append('\\').Append(ch); + continue; + } + switch (ch) + { + case '\t': + sb.Append(' '); + break; + case '\n': + sb.Append('\n'); + break; + case < ' ' or '': + break; + default: + sb.Append(ch); + break; + } + } + return sb.ToString(); + } + + /// A line that would start a heading, list item or term gets its first character escaped. + private static string LineStart(string line) + { + if (line.Length == 0) + return line; + if (line[0] is '=' or '-' or '+' or '/') + return "\\" + line; + var digits = line.TakeWhile(char.IsAsciiDigit).Count(); + return digits > 0 && digits < line.Length && line[digits] == '.' && (digits + 1 == line.Length || char.IsWhiteSpace(line[digits + 1])) + ? line[..digits] + "\\" + line[digits..] + : line; + } +} diff --git a/src/Filee.Engines/Hwp/Hwpx/TexMathWriter.cs b/src/Filee.Engines/Hwp/Hwpx/TexMathWriter.cs new file mode 100644 index 000000000..cff720dcc --- /dev/null +++ b/src/Filee.Engines/Hwp/Hwpx/TexMathWriter.cs @@ -0,0 +1,516 @@ +// The equation tree (MathNode) → TeX math, for the writers whose formats hold TeX: LaTeX ($…$ and \[…\]), Markdown +// ($…$, which MarkdownReader reads back through LatexMath) and AsciiDoc (latexmath). Equations read from TeX keep their +// source instead; this is for Word math (OMML), 한글 equations and MathML, which come as a tree. +// +// What it writes stays within amsmath and amssymb: symbols by the command LatexMath reads them from (≤ → \leq), large +// operators with limits, \frac / \genfrac, \sqrt[n], scripts (primes as '), \lim, function names (\sin, else +// \operatorname), matrix / pmatrix / bmatrix / vmatrix, cases, aligned / gathered / array for piles, \left … \right, +// accents (wide ones over several characters), \mathrm \mathit \mathbf \boldsymbol, the letters of \mathbb \mathcal +// \mathfrak, spaces, \not, and \text{…} for +// words and characters without a command. Greek capitals that look Latin are Latin letters, as TeX has no command for +// them. A character with no command at all is written as itself inside \text, which XeLaTeX and LuaLaTeX set. +// +// The tree's depth is bounded by its readers (EquationParser, LatexMath, OmmlReader: 200 at most); a tree deeper than +// MaxDepth is not written (null), and the caller keeps the equation's text. + +using System.Globalization; +using System.Text; + +namespace Filee.Engines.Hwp.Hwpx; + +internal static class TexMathWriter +{ + /// Levels of a tree written; the walk recurses per level. + public const int MaxDepth = 256; + + /// Function names LaTeX has a command for. + private static readonly HashSet Functions = new(StringComparer.Ordinal) + { + "sin", "cos", "tan", "cot", "sec", "csc", "arcsin", "arccos", "arctan", "sinh", "cosh", "tanh", "coth", "log", + "ln", "lg", "exp", "det", "dim", "ker", "hom", "arg", "gcd", "max", "min", "sup", "inf", "limsup", "liminf", + "Pr", "deg", + }; + + /// Large operators by their character. + private static readonly Dictionary BigOperators = new(StringComparer.Ordinal) + { + ["∑"] = "sum", + ["∏"] = "prod", + ["∐"] = "coprod", + ["⋃"] = "bigcup", + ["⋂"] = "bigcap", + ["⋁"] = "bigvee", + ["⋀"] = "bigwedge", + ["⨁"] = "bigoplus", + ["⨂"] = "bigotimes", + ["⨀"] = "bigodot", + ["⨄"] = "biguplus", + ["∫"] = "int", + ["∬"] = "iint", + ["∭"] = "iiint", + ["∮"] = "oint", + }; + + /// Characters whose TeX command LatexMath doesn't take, or whose command needs another package. + private static readonly Dictionary Extra = new(StringComparer.Ordinal) + { + ["⋅"] = "\\cdot", + ["∙"] = "\\bullet", + ["−"] = "-", + ["′"] = "\\prime", + ["°"] = "^{\\circ}", + ["≔"] = ":=", + ["≕"] = "=:", + ["≢"] = "\\not\\equiv", + ["ο"] = "o", + ["∌"] = "\\not\\ni", + ["⊄"] = "\\not\\subset", + ["⊅"] = "\\not\\supset", + ["≉"] = "\\not\\approx", + ["⟨"] = "\\langle", + ["⟩"] = "\\rangle", + ["〈"] = "\\langle", + ["〉"] = "\\rangle", + }; + + /// Greek capitals TeX writes as the Latin letter they look like. + private const string GreekLookingLatin = "ΑAΒBΕEΖZΗHΙIΚKΜMΝNΟOΡPΤTΧX"; + + /// The TeX of (without $ delimiters), or null when it nests deeper than . + public static string? Write(MathNode node) + { + if (EquationScript.Deeper(node, MaxDepth)) + return null; + var writer = new Writer(); + writer.Node(node, aligned: false); + return writer.ToString().Trim(); + } + + private sealed class Writer + { + private readonly StringBuilder _sb = new(); + + /// The last thing written is a control word: a letter after it needs a space. + private bool _word; + + public override string ToString() => _sb.ToString(); + + private void Word(string name) + { + _sb.Append('\\').Append(name); + _word = true; + } + + private void Raw(string text) + { + if (text.Length == 0) + return; + if (_word && char.IsAsciiLetter(text[0])) + _sb.Append(' '); + _sb.Append(text); + _word = text.Length > 1 && text[0] == '\\' && text.Skip(1).All(char.IsAsciiLetter); + } + + private void Group(MathNode node, bool aligned = false) + { + Raw("{"); + Node(node, aligned); + Raw("}"); + } + + /// Inside an aligned pile, where an alignment mark is an &. + public void Node(MathNode node, bool aligned) + { + switch (node) + { + case MathRow row: + foreach (var item in row.Items) + Node(item, aligned); + break; + case MathText text: + Characters(text.Text); + break; + case MathNumber number: + Characters(number.Digits); + break; + case MathSymbol symbol: + Characters(symbol.Text); + break; + case MathCommand command: + Characters(command.Glyph); + break; + case MathQuoted quoted: + Text(quoted.Text); + break; + case MathFunction function: + if (Functions.Contains(function.Name)) + Word(function.Name); + else if (function.Name == "mod") + Word("bmod"); + else + { + Word("operatorname"); + Raw("{"); + Text(function.Name, plain: true); + Raw("}"); + } + break; + case MathFraction fraction: + if (fraction.Bar) + Word("frac"); + else + Raw("\\genfrac{}{}{0pt}{}"); + Group(fraction.Numerator); + Group(fraction.Denominator); + break; + case MathRoot root: + Word("sqrt"); + if (root.Index is not null) + { + Raw("["); + Node(root.Index, aligned: false); + Raw("]"); + } + Group(root.Body); + break; + case MathScripts scripts: + Base(scripts.Base); + Scripts(scripts.Sub, scripts.Sup); + break; + case MathBigOperator big: + if (BigOperators.TryGetValue(big.Glyph, out var name)) + Word(name); + else + { + Raw("\\mathop{"); + Text(big.Glyph); + Raw("}"); + } + Scripts(big.Lower, big.Upper); + break; + case MathLimit limit: + Word("lim"); + Scripts(limit.Lower, null); + break; + case MathMatrix matrix: + Environment(matrix.Command switch + { + "pmatrix" => "pmatrix", + "bmatrix" => "bmatrix", + "dmatrix" or "vmatrix" => "vmatrix", + _ => "matrix", + }, null, matrix.Rows); + break; + case MathCases cases: + Environment("cases", null, cases.Rows); + break; + case MathPile pile: + Pile(pile); + break; + case MathBrackets { Left: "(", Right: ")", Body: MathFraction { Bar: false } binomial }: + Word("binom"); + Group(binomial.Numerator); + Group(binomial.Denominator); + break; + case MathBrackets brackets: + Word("left"); + Delimiter(brackets.Left); + Node(brackets.Body, aligned); + Word("right"); + Delimiter(brackets.Right); + break; + case MathAccent accent: + Accent(accent); + break; + case MathStyled styled: + Styled(styled); + break; + case MathSpace space: + Space(space.Width); + break; + case MathAlign: + if (aligned) + Raw("&"); + break; + } + } + + /// The base of scripts: braced unless it is one item TeX takes as it is. + private void Base(MathNode node) + { + var single = node is MathRow { Items.Count: 1 } row ? row.Items[0] : node; + if (single is MathText { Text.Length: 1 } or MathNumber or MathSymbol { Text.Length: 1 } or MathCommand or MathFunction + or MathBrackets or MathAccent or MathStyled or MathRoot or MathFraction or MathLimit) + Node(single, aligned: false); + else + Group(node); + } + + private void Scripts(MathNode? sub, MathNode? sup) + { + // f', f'': a superscript of primes only. + if (sup is not null && Primes(sup) is var primes and > 0) + { + Raw(new string('\'', primes)); + sup = null; + } + if (sub is not null) + { + Raw("_"); + Group(sub); + } + if (sup is not null) + { + Raw("^"); + Group(sup); + } + } + + private static int Primes(MathNode node) + { + var items = node is MathRow row ? row.Items : [node]; + return items.Count > 0 && items.All(i => i is MathCommand { Glyph: "′" } or MathSymbol { Text: "′" or "'" }) ? items.Count : 0; + } + + private void Environment(string name, string? columns, IReadOnlyList> rows) + { + Raw("\\begin{" + name + "}"); + if (columns is not null) + Raw("{" + columns + "}"); + for (var r = 0; r < rows.Count; r++) + { + if (r > 0) + Raw(" \\\\ "); + for (var c = 0; c < rows[r].Count; c++) + { + if (c > 0) + Raw(" & "); + Node(rows[r][c], aligned: false); + } + } + Raw("\\end{" + name + "}"); + } + + /// Lines with alignment marks as aligned (the marks as &); others gathered, or as an array to the left or right. + private void Pile(MathPile pile) + { + var marks = pile.Rows.Any(r => r is MathRow row && row.Items.Any(i => i is MathAlign)); + var (name, columns) = marks ? ("aligned", (string?)null) : pile.Align switch + { + "lpile" => ("array", "l"), + "rpile" => ("array", "r"), + _ => ("gathered", null), + }; + Raw("\\begin{" + name + "}"); + if (columns is not null) + Raw("{" + columns + "}"); + for (var r = 0; r < pile.Rows.Count; r++) + { + if (r > 0) + Raw(" \\\\ "); + Node(pile.Rows[r], aligned: marks); + } + Raw("\\end{" + name + "}"); + } + + private void Delimiter(string bracket) + { + switch (bracket) + { + case "": + Raw("."); + break; + case "(" or ")" or "[" or "]" or "|" or "/": + Raw(bracket); + break; + case "{" or "}": + Raw("\\" + bracket); + break; + case "‖": + Raw("\\|"); + break; + case "⟨" or "〈": + Word("langle"); + break; + case "⟩" or "〉": + Word("rangle"); + break; + case "⌈": + Word("lceil"); + break; + case "⌉": + Word("rceil"); + break; + case "⌊": + Word("lfloor"); + break; + case "⌋": + Word("rfloor"); + break; + default: + // Not a delimiter TeX stretches: no bracket, and the character as it is. + Raw("."); + Characters(bracket); + break; + } + } + + private void Accent(MathAccent accent) + { + var body = accent.Body is MathRow { Items.Count: 1 } row ? row.Items[0] : accent.Body; + var single = body is MathText { Text.Length: 1 } or MathCommand; + string? command = accent.Command switch + { + "hat" => single ? "hat" : "widehat", + "tilde" => single ? "tilde" : "widetilde", + "check" or "acute" or "grave" or "dot" or "ddot" or "breve" => accent.Command, + "bar" => single ? "bar" : "overline", + "OVERLINE" or "overline" => "overline", + "under" or "UNDERLINE" or "underline" => "underline", + "vec" => single ? "vec" : "overrightarrow", + "dyad" => "overleftrightarrow", + _ => null, + }; + if (command is not null) + { + Word(command); + Group(accent.Body); + return; + } + switch (accent.Command) + { + case "arch": + Raw("\\overset{\\frown}"); + Group(accent.Body); + break; + case "not": + Word("not"); + if (body is MathSymbol or MathCommand) + Node(body, aligned: false); + else + Group(accent.Body); + break; + default: + Node(accent.Body, aligned: false); + break; + } + } + + private void Styled(MathStyled styled) + { + switch (styled.Command) + { + case "rm": + Word("mathrm"); + Group(styled.Body); + break; + case "it": + Word("mathit"); + Group(styled.Body); + break; + case "bold" when styled.Body is MathStyled { Command: "rm" } upright: + Word("mathbf"); + Group(upright.Body); + break; + case "bold": + Word("boldsymbol"); + Group(styled.Body); + break; + default: + Node(styled.Body, aligned: false); + break; + } + } + + /// A space of (1 is half the font size, see MathLayout). + private void Space(double width) + { + var known = new (double Width, string Tex)[] + { + (1 / 3.0, "\\,"), (4 / 9.0, "\\:"), (5 / 9.0, "\\;"), (2 / 3.0, "\\ "), (1, "\\enspace"), (2, "\\quad"), (4, "\\qquad"), + }; + foreach (var (w, tex) in known) + { + if (Math.Abs(w - width) < 0.01) + { + Raw(tex); + return; + } + } + if (width <= 0) + return; + Raw(string.Create(CultureInfo.InvariantCulture, $"\\hspace{{{width * 0.5:0.###}em}}")); + } + + /// + /// Characters of a variable, number or symbol: ASCII letters and digits as they are, TeX's specials escaped, + /// symbols by their command, letters without one (CJK) in \text. + /// + private void Characters(string text) + { + var pending = new StringBuilder(); + void Flush() + { + if (pending.Length > 0) + Text(pending.ToString()); + pending.Clear(); + } + + foreach (var rune in text.EnumerateRunes()) + { + var s = rune.ToString(); + var c = rune.IsBmp ? (char)rune.Value : '\0'; + string? tex = c switch + { + _ when char.IsAsciiLetterOrDigit(c) => s, + '+' or '-' or '=' or '<' or '>' or '(' or ')' or '[' or ']' or ',' or '.' or '/' or '|' or '!' or '*' or ':' or ';' or '?' or '\'' or '@' => s, + '{' or '}' or '#' or '$' or '%' or '&' or '_' => "\\" + s, + '\\' => "\\backslash", + '~' => "\\sim", + '^' => "\\hat{}", + '"' => "\\text{\"}", + ' ' => "\\ ", + _ when GreekLookingLatin.IndexOf(c) is var at and >= 0 && at % 2 == 0 => GreekLookingLatin[at + 1].ToString(), + _ when Extra.TryGetValue(s, out var extra) => extra, + _ when LatexMath.SymbolCommand(s) is { } command => "\\" + command, + _ when LatexMath.AlphabetLetter(s) is { } letter => $"\\{letter.Command}{{{letter.Letter}}}", + _ => null, + }; + if (tex is null) + { + pending.Append(s); + continue; + } + Flush(); + if (tex.StartsWith('\\') && tex.Skip(1).All(char.IsAsciiLetter)) + Word(tex[1..]); + else + Raw(tex); + } + Flush(); + } + + /// Text (\text{…}) with the characters TeX's text mode treats specially escaped. + private void Text(string text, bool plain = false) + { + if (text.Length == 0) + return; + var escaped = new StringBuilder(text.Length + 8); + foreach (var c in text) + { + escaped.Append(c switch + { + '\\' => "\\textbackslash{}", + '{' or '}' or '#' or '$' or '%' or '&' or '_' => "\\" + c, + '~' => "\\textasciitilde{}", + '^' => "\\textasciicircum{}", + _ => c.ToString(), + }); + } + if (plain) + Raw(escaped.ToString()); + else + Raw("\\text{" + escaped + "}"); + } + } +} diff --git a/src/Filee.Engines/Hwp/Hwpx/TypstMathWriter.cs b/src/Filee.Engines/Hwp/Hwpx/TypstMathWriter.cs new file mode 100644 index 000000000..ea7477bf2 --- /dev/null +++ b/src/Filee.Engines/Hwp/Hwpx/TypstMathWriter.cs @@ -0,0 +1,479 @@ +// The equation tree (MathNode) → Typst math, for the Typst writer ($ … $). Typst's math is its own language, so even +// an equation read from TeX is written from its tree. +// +// Atoms are separated by spaces (Typst draws math spacing from the atoms, not the spaces, and a space keeps letters +// from running into one multi-letter name or shorthands like "->" from forming). Symbols are their Unicode characters, +// which Typst's math takes as they are; the characters with a meaning in its syntax are escaped (\( \, \_ …), words +// and CJK text are quoted. Fractions frac(a, b), roots sqrt / root, scripts x_(i)^(n) (attach() for a base of several +// atoms, primes as '), large operators with their limits, function names as op("…") (with limits for lim, max …), +// matrices mat(delim: …), cases(…), piles as lines (\) with & marks at the top or as a matrix inside, stretched +// brackets ( … ) or lr(…), abs / norm / floor / ceil, accents (hat, tilde, overline, arrow …), upright / italic / +// bold, spaces (thin, med, thick, quad, wide or #h). Cells and arguments are separated by the writer's own commas, so +// a comma or semicolon in their content is escaped. +// +// A tree deeper than MaxDepth is not written (null), as in TexMathWriter. + +using System.Globalization; +using System.Text; + +namespace Filee.Engines.Hwp.Hwpx; + +internal static class TypstMathWriter +{ + /// + public const int MaxDepth = TexMathWriter.MaxDepth; + + /// Operator names Typst sets with their limits below in display math. + private static readonly HashSet WithLimits = new(StringComparer.Ordinal) + { + "lim", "liminf", "limsup", "max", "min", "sup", "inf", "det", "gcd", "lcm", "Pr", + }; + + /// The Typst math of (without $ delimiters), or null when it nests deeper than . + public static string? Write(MathNode node) + { + if (EquationScript.Deeper(node, MaxDepth)) + return null; + var writer = new Writer(); + // Lines of the whole equation are lines; a pile inside something else is a matrix. + if (node is MathPile pile) + writer.Lines(pile); + else if (node is MathRow { Items: [MathPile first, ..] } row && row.Items.Skip(1).All(i => i is MathSpace or MathQuoted)) + { + // Lines with their ag after them. + writer.Lines(first); + foreach (var item in row.Items.Skip(1)) + writer.Node(item); + } + else + writer.Node(node); + return writer.ToString().Trim(); + } + + private sealed class Writer + { + private readonly StringBuilder _sb = new(); + + /// Calls the writer is inside: their commas and semicolons separate arguments. + private int _arguments; + + public override string ToString() => _sb.ToString(); + + /// A space before the next atom, unless it follows an opening bracket or another space. + private void Separate() + { + // (an escaped "\(" is an atom, not an opening) + if (_sb.Length == 0 || _sb[^1] is ' ' or '\n' || (_sb[^1] == '(' && (_sb.Length < 2 || _sb[^2] != '\\'))) + return; + _sb.Append(' '); + } + + private void Atom(string text) + { + Separate(); + _sb.Append(text); + } + + /// A call: name( arguments separated by ", " ). + private void Call(string name, params MathNode[] arguments) + { + Atom(name); + _sb.Append('('); + _arguments++; + for (var i = 0; i < arguments.Length; i++) + { + if (i > 0) + _sb.Append(", "); + Content(arguments[i]); + } + _arguments--; + _sb.Append(')'); + } + + /// Content of an argument; empty content as "" (an argument can't be left blank). + private void Content(MathNode node) + { + var start = _sb.Length; + Node(node); + if (_sb.ToString(start, _sb.Length - start).Trim().Length == 0) + { + _sb.Length = start; + _sb.Append("\"\""); + } + } + + public void Lines(MathPile pile) + { + for (var r = 0; r < pile.Rows.Count; r++) + { + if (r > 0) + _sb.Append(" \\\n"); + Node(pile.Rows[r], aligned: true); + } + } + + /// At the top of the equation, where an alignment mark is an &. + public void Node(MathNode node, bool aligned = false) + { + switch (node) + { + case MathRow row: + foreach (var item in row.Items) + Node(item, aligned); + break; + case MathText text: + Letters(text.Text); + break; + case MathNumber number: + if (number.Digits.All(c => char.IsAsciiDigit(c) || c == '.')) + Atom(number.Digits); + else + Symbols(number.Digits); + break; + case MathSymbol symbol: + Symbols(symbol.Text); + break; + case MathCommand command: + Symbols(command.Glyph); + break; + case MathQuoted quoted: + if (quoted.Text.Length > 0) + Atom(Quoted(quoted.Text)); + break; + case MathFunction function: + Atom(Operator(function.Name)); + break; + case MathFraction { Bar: true } fraction: + Call("frac", fraction.Numerator, fraction.Denominator); + break; + case MathFraction stack: + Matrix("#none", [[stack.Numerator], [stack.Denominator]]); + break; + case MathRoot { Index: null } root: + Call("sqrt", root.Body); + break; + case MathRoot root: + Call("root", root.Index, root.Body); + break; + case MathScripts scripts: + Scripts(scripts.Base, scripts.Sub, scripts.Sup); + break; + case MathBigOperator big: + Symbols(big.Glyph); + Attachments(big.Lower, big.Upper); + break; + case MathLimit limit: + Atom("lim"); + Attachments(limit.Lower, null); + break; + case MathMatrix matrix: + Matrix(matrix.Command switch + { + "pmatrix" => "\"(\"", + "bmatrix" => "\"[\"", + "dmatrix" or "vmatrix" => "\"|\"", + _ => "#none", + }, matrix.Rows); + break; + case MathCases cases: + Atom("cases"); + _sb.Append('('); + _arguments++; + for (var r = 0; r < cases.Rows.Count; r++) + { + if (r > 0) + _sb.Append(", "); + for (var c = 0; c < cases.Rows[r].Count; c++) + { + if (c > 0) + _sb.Append(" &"); + Node(cases.Rows[r][c]); + } + } + _arguments--; + _sb.Append(')'); + break; + case MathPile pile: + Matrix("#none", [.. pile.Rows.Select(Cells)]); + break; + case MathBrackets { Left: "(", Right: ")", Body: MathFraction { Bar: false } binomial }: + Call("binom", binomial.Numerator, binomial.Denominator); + break; + case MathBrackets brackets: + Brackets(brackets); + break; + case MathAccent accent: + Accent(accent); + break; + case MathStyled styled: + switch (styled.Command) + { + case "rm": + Call("upright", styled.Body); + break; + case "it": + Call("italic", styled.Body); + break; + case "bold": + Call("bold", styled.Body); + break; + default: + Node(styled.Body, aligned); + break; + } + break; + case MathSpace space: + Space(space.Width); + break; + case MathAlign: + if (aligned) + Atom("&"); + break; + } + } + + private void Scripts(MathNode baseNode, MathNode? sub, MathNode? sup) + { + var single = baseNode is MathRow { Items.Count: 1 } row ? row.Items[0] : baseNode; + var atom = single is MathText { Text.Length: 1 } or MathNumber or MathSymbol { Text.Length: 1 } or MathCommand or MathFunction + or MathQuoted or MathFraction or MathRoot or MathBrackets or MathAccent or MathStyled or MathLimit or MathMatrix; + if (atom) + { + Node(single); + Attachments(sub, sup); + return; + } + // A base of several atoms (or none): attach() keeps it one. + Atom("attach"); + _sb.Append('('); + _arguments++; + Content(baseNode); + if (sub is not null) + { + _sb.Append(", b: "); + Content(sub); + } + if (sup is not null) + { + _sb.Append(", t: "); + Content(sup); + } + _arguments--; + _sb.Append(')'); + } + + /// _( … ) and ^( … ) right after the base (primes as '). + private void Attachments(MathNode? sub, MathNode? sup) + { + if (sup is not null && Primes(sup) is var primes and > 0) + { + _sb.Append('\'', primes); + sup = null; + } + if (sub is not null) + { + _sb.Append("_("); + _arguments++; + Content(sub); + _arguments--; + _sb.Append(')'); + } + if (sup is not null) + { + _sb.Append("^("); + _arguments++; + Content(sup); + _arguments--; + _sb.Append(')'); + } + } + + private static int Primes(MathNode node) + { + var items = node is MathRow row ? row.Items : [node]; + return items.Count > 0 && items.All(i => i is MathCommand { Glyph: "′" } or MathSymbol { Text: "′" or "'" }) ? items.Count : 0; + } + + private void Matrix(string delimiter, IReadOnlyList> rows) + { + Atom("mat"); + _sb.Append("(delim: ").Append(delimiter); + _arguments++; + for (var r = 0; r < rows.Count; r++) + { + _sb.Append(r == 0 ? ", " : "; "); + for (var c = 0; c < rows[r].Count; c++) + { + if (c > 0) + _sb.Append(", "); + Content(rows[r][c]); + } + } + _arguments--; + _sb.Append(')'); + } + + /// A line of a pile split at its alignment marks. + private static IReadOnlyList Cells(MathNode line) + { + if (line is not MathRow row || !row.Items.Any(i => i is MathAlign)) + return [line]; + var cells = new List(); + var cell = new List(); + foreach (var item in row.Items) + { + if (item is MathAlign) + { + cells.Add(new MathRow(cell)); + cell = []; + } + else + { + cell.Add(item); + } + } + cells.Add(new MathRow(cell)); + return cells; + } + + private void Brackets(MathBrackets brackets) + { + string? function = (brackets.Left, brackets.Right) switch + { + ("|", "|") => "abs", + ("‖", "‖") => "norm", + ("⌊", "⌋") => "floor", + ("⌈", "⌉") => "ceil", + _ => null, + }; + if (function is not null) + { + Call(function, brackets.Body); + return; + } + if ((brackets.Left, brackets.Right) is ("(", ")") or ("[", "]") or ("{", "}")) + { + // Matched brackets stretch by themselves. + Atom(brackets.Left); + Node(brackets.Body); + _sb.Append(brackets.Right); + return; + } + Atom("lr"); + _sb.Append('('); + _arguments++; + if (brackets.Left.Length > 0) + Symbols(brackets.Left); + Node(brackets.Body); + if (brackets.Right.Length > 0) + Symbols(brackets.Right); + _arguments--; + _sb.Append(')'); + } + + private void Accent(MathAccent accent) + { + var body = accent.Body is MathRow { Items.Count: 1 } row ? row.Items[0] : accent.Body; + var single = body is MathText { Text.Length: 1 } or MathCommand; + string? function = accent.Command switch + { + "hat" or "tilde" or "acute" or "grave" or "dot" or "breve" => accent.Command, + "check" => "caron", + "ddot" => "dot.double", + "bar" => single ? "macron" : "overline", + "OVERLINE" or "overline" => "overline", + "under" or "UNDERLINE" or "underline" => "underline", + "vec" => "arrow", + _ => null, + }; + if (function is not null) + { + Call(function, accent.Body); + return; + } + switch (accent.Command) + { + case "dyad" or "arch": + Atom("accent"); + _sb.Append('('); + _arguments++; + Content(accent.Body); + _arguments--; + _sb.Append(accent.Command == "dyad" ? ", arrow.l.r)" : ", ⌢)"); + break; + case "not" when body is MathSymbol { Text.Length: 1 } or MathCommand { Glyph.Length: 1 }: + { + var glyph = body is MathSymbol symbol ? symbol.Text : ((MathCommand)body).Glyph; + Atom(Escape(glyph) + "̸"); + break; + } + case "not": + Call("cancel", accent.Body); + break; + default: + Node(accent.Body); + break; + } + } + + private void Space(double width) + { + var name = width switch + { + _ when Math.Abs(width - 1 / 3.0) < 0.01 => "thin", + _ when Math.Abs(width - 4 / 9.0) < 0.01 => "med", + _ when Math.Abs(width - 5 / 9.0) < 0.01 => "thick", + _ when Math.Abs(width - 2) < 0.01 => "quad", + _ when Math.Abs(width - 4) < 0.01 => "wide", + _ => null, + }; + if (name is not null) + Atom(name); + else if (width > 0) + Atom(string.Create(CultureInfo.InvariantCulture, $"#h({width * 0.5:0.###}em)")); + } + + /// + /// A variable: Latin and Greek letters each an atom of its own (several together would be one name), other + /// writing (CJK) as quoted text. + /// + private void Letters(string text) + { + if (text.EnumerateRunes().Any(r => Rune.IsLetter(r) && r.Value > 0x3FF)) + { + Atom(Quoted(text)); + return; + } + Symbols(text); + } + + /// Each character an atom, those with a meaning in Typst's syntax escaped. + private void Symbols(string text) + { + foreach (var rune in text.EnumerateRunes()) + { + if (Rune.IsWhiteSpace(rune)) + continue; + Atom(Escape(rune.ToString())); + } + } + + /// + /// A character as an atom: escaped when it means something in Typst's syntax (a comma or semicolon only inside + /// an argument, where it would separate). + /// + private string Escape(string character) => + character is "\\" or "/" or "_" or "^" or "&" or "#" or "$" or "\"" or "(" or ")" or "[" or "]" or "{" or "}" or "'" or "@" + || (_arguments > 0 && character is "," or ";") + ? "\\" + character + : character; + + private static string Quoted(string text) => + "\"" + text.Replace("\\", "\\\\", StringComparison.Ordinal).Replace("\"", "\\\"", StringComparison.Ordinal) + "\""; + + private static string Operator(string name) => + name == "lim" ? "lim" + : WithLimits.Contains(name) ? $"op({Quoted(name)}, limits: #true)" + : $"op({Quoted(name)})"; + } +} diff --git a/src/Filee.Engines/Text/MarkupConverter.cs b/src/Filee.Engines/Text/MarkupConverter.cs index 2504d7a70..c5e04e205 100644 --- a/src/Filee.Engines/Text/MarkupConverter.cs +++ b/src/Filee.Engines/Text/MarkupConverter.cs @@ -2,7 +2,8 @@ // built-in reader reads (DocumentReaders) → each of them, and each of them → Markdown, HTML and plain text. Their // readers are in DocumentReaders, so the HWPX, DOCX and ODT writers and the Filee renderer take them too (→ HWPX, DOCX, // ODT, PDF). The edges cost 8 like the DOCX writer's, below Pandoc's (10, and 11 into a markup language), so the -// built-in route wins; Pandoc keeps what only it does (LaTeX, Typst, AsciiDoc, man pages, slides, RTF and EPUB out). +// built-in route wins. LaTeX, AsciiDoc, Typst and man pages are written here too (WriteOnly), from every built-in +// reader's format; reading them stays Pandoc's, as do slides, and RTF and EPUB straight from a markup language. // Pictures of a text result go into a "_files" folder next to it (MarkdownPictures), a notebook embeds its own. // Mind maps (MindMapReader), read only, are written as Markdown, HTML and text here too; into OPML every topic is a // heading, so the outline keeps its whole tree. @@ -21,7 +22,8 @@ namespace Filee.Engines.Text; /// Maps a picture file to the link written into the result, or null to leave it out. /// Scratch folder of the job (pictures converted to a common format). /// The preset leaves pictures out. -internal sealed record MarkupWriteContext(Func Pictures, string WorkFolder, bool LeaveOutPictures); +/// Stops the writer between blocks. +internal sealed record MarkupWriteContext(Func Pictures, string WorkFolder, bool LeaveOutPictures, CancellationToken Cancellation = default); /// Reads and writes the markup languages Filee owns. public sealed class MarkupConverter : IConverter @@ -42,6 +44,19 @@ public sealed class MarkupConverter : IConverter ["dbk"] = (document, context) => DocBookWriter.Write(document, context.Pictures), }; + /// + /// Languages written but not read here (macro and scripting languages, which Pandoc reads): format id → the text of + /// the file. Their edges come from every built-in reader's format, like the writers'. + /// + internal static readonly IReadOnlyDictionary> WriteOnly = + new Dictionary>(StringComparer.OrdinalIgnoreCase) + { + ["tex"] = (document, context) => new LatexWriter(context.Pictures) { Cancellation = context.Cancellation }.Write(document), + ["adoc"] = (document, context) => new AsciiDocWriter(context.Pictures) { Cancellation = context.Cancellation }.Write(document), + ["typ"] = (document, context) => new TypstWriter(context.Pictures) { Cancellation = context.Cancellation }.Write(document), + ["man"] = (document, context) => new ManWriter { Cancellation = context.Cancellation }.Write(document), + }; + /// Formats each markup language is also written as (the readers' other outputs are the writers' edges). private static readonly string[] TextTargets = ["md", "html", "txt"]; @@ -55,7 +70,7 @@ public sealed class MarkupConverter : IConverter /// public IReadOnlyList Edges { get; } = [ - .. from to in Writers.Keys + .. from to in Writers.Keys.Concat(WriteOnly.Keys) from source in DocumentReaders.BuiltInFormats where source != to select new ConversionEdge(source, to, source == "pdf" ? Cost + DocumentReaders.PdfPenalty : Cost, DirectOnly: true), @@ -65,7 +80,7 @@ from to in TextTargets ]; public EngineStatus GetStatus() => - EngineStatus.Available(string.Join(", ", Writers.Keys.Select(f => FormatRegistry.Get(f).DisplayName)), EngineVersions.BuiltIn); + EngineStatus.Available(string.Join(", ", Writers.Keys.Concat(WriteOnly.Keys).Select(f => FormatRegistry.Get(f).DisplayName)), EngineVersions.BuiltIn); public async Task> ConvertAsync(ConversionStep step, IProgress? progress, CancellationToken cancellationToken) { @@ -80,16 +95,16 @@ public async Task> ConvertAsync(ConversionStep step, IProg var output = step.Output.Allocate(FormatRegistry.Get(step.To).PrimaryExtension); if (output is null) return []; - await Task.Run(() => Write(document, step, output), cancellationToken); + await Task.Run(() => Write(document, step, output, cancellationToken), cancellationToken); progress?.Report(1); return [output]; } - private static void Write(HDocument document, ConversionStep step, string output) + private static void Write(HDocument document, ConversionStep step, string output, CancellationToken cancellationToken) { var leaveOut = step.Preset.Document.LeaveOutPictures; Func pictures = leaveOut ? _ => null : new MarkdownPictures(output, step.Output, step.WorkDirectory).Add; - if (step.To is "md" || Writers.ContainsKey(step.To)) + if (step.To is "md" || Writers.ContainsKey(step.To) || WriteOnly.ContainsKey(step.To)) MarkupBlocks.OpenCodeBoxes(document); // Markdown's code boxes are code blocks in markup switch (step.To) { @@ -110,9 +125,14 @@ private static void Write(HDocument document, ConversionStep step, string output File.WriteAllText(output, PlainTextWriter.Write(document), new UTF8Encoding(encoderShouldEmitUTF8Identifier: true)); break; default: - File.WriteAllText(output, Writers[step.To](document, new MarkupWriteContext(pictures, step.WorkDirectory, leaveOut)), - new UTF8Encoding(false)); - break; + { + var context = new MarkupWriteContext(pictures, step.WorkDirectory, leaveOut, cancellationToken); + var text = WriteOnly.TryGetValue(step.To, out var writeOnly) ? writeOnly(document, context) : Writers[step.To](document, context); + // Through a temporary file: a failed write leaves nothing under the output's name. + var bytes = new UTF8Encoding(false).GetBytes(text); + WriterTempFile.Write(output, stream => stream.Write(bytes)); + break; + } } } diff --git a/tests/Filee.Engines.Tests/MacroMarkupWriterTests.cs b/tests/Filee.Engines.Tests/MacroMarkupWriterTests.cs new file mode 100644 index 000000000..4529e34f2 --- /dev/null +++ b/tests/Filee.Engines.Tests/MacroMarkupWriterTests.cs @@ -0,0 +1,500 @@ +// The built-in writers of LaTeX, AsciiDoc, Typst and man pages (Hwp/Hwpx/Markup), which Pandoc wrote before: one +// model built in code with everything they write (headings, emphasis, nested and numbered lists, quotes, code, a table +// with merged cells, links, footnotes, pictures, equations and every special character), the text each writes for it, +// the structure that keeps the file readable (balanced braces and environments, requests, blocks), their routes, which +// now win over Pandoc's, and cancellation and limits. + +using System.Text; +using System.Text.RegularExpressions; +using Filee.Core.Conversion; +using Filee.Core.Presets; +using Filee.Engines.Documents; +using Filee.Engines.Hwp.Hwpx; +using Filee.Engines.Hwp.Hwpx.Markup; +using Filee.Engines.Text; + +namespace Filee.Engines.Tests; + +public sealed partial class MacroMarkupWriterTests(EngineFixture fx) : IClassFixture +{ + /// Text with every character one of the four languages treats specially. + private const string Specials = @"a\b {c} $d & #e _f %g ~h ^i |k| ""l"" [m] n--o '' `p` q@r *s* t+u :: v;; w... x://y (C) -? //z"; + + private static HText Text(string text, HCharFormat format = default) => new(text, format); + + private static HParagraph Paragraph(params HInline[] inlines) + { + var paragraph = new HParagraph(); + paragraph.Inlines.AddRange(inlines); + return paragraph; + } + + private static HCell Cell(string text, int rowSpan = 1, int colSpan = 1) + { + var cell = new HCell { RowSpan = rowSpan, ColSpan = colSpan }; + cell.Blocks.Add(Paragraph(Text(text))); + return cell; + } + + private static HParagraph Item(HNumbering numbering, int level, string text, bool numbered = true) + { + var paragraph = Paragraph(Text(text)); + paragraph.List = new HListRef(numbering, level, numbered); + return paragraph; + } + + private static HDocument Document(params HBlock[] blocks) + { + var document = new HDocument(); + var section = new HSection(); + section.Blocks.AddRange(blocks); + document.Sections.Add(section); + return document; + } + + /// A document with everything the writers write. + private static HDocument Sample() + { + var numbering = new HNumbering(); + numbering.Levels.Add(new HNumberingLevel("DIGIT", "•", 1, Bullet: true)); + numbering.Levels.Add(new HNumberingLevel("DIGIT", "^2.", 3, Bullet: false)); + numbering.Levels.Add(new HNumberingLevel("DIGIT", "◦", 1, Bullet: true)); + + var note = new HNote(endnote: false); + note.Blocks.Add(Paragraph(Text("The note text."))); + var link = new HLink("https://example.com/a b#c"); + link.Content.Add(Text("the site")); + var fraction = new MathFraction(new MathText("a"), new MathText("b")); + var display = new HParagraph { Format = new HParaFormat(Align: HAlign.Center) }; + display.Inlines.Add(new HEquation(EquationParser.Parse("SUM _{i=1} ^{n} i"), default)); + + var table = new HTable { ColumnCount = 3 }; + var header = new HRow { Header = true }; + header.Cells.AddRange([Cell("H1"), Cell("H2"), Cell("H3")]); + var first = new HRow(); + first.Cells.AddRange([Cell("wide", colSpan: 2), Cell("tall", rowSpan: 2)]); + var second = new HRow(); + second.Cells.AddRange([Cell("x"), Cell("y")]); + table.Rows.AddRange([header, first, second]); + + var document = Document( + new HParagraph { HeadingLevel = 1, Inlines = { Text("Intro_1") } }, + Paragraph(Text("Plain "), Text("bold", new(Bold: true)), Text(" "), Text("italic", new(Italic: true)), Text(" "), + Text("under", new(Underline: true)), Text(" "), Text("gone", new(Strike: true)), Text(" "), + Text("code()", new(Shade: HtmlReader.CodeShade)), Text(" E=mc"), Text("2", new(Superscript: true)), Text(" H"), + Text("2", new(Subscript: true)), Text("O"), new HLineBreak(default), Text("next line")), + Paragraph(Text(Specials)), + Paragraph(Text("See "), link, Text(" and a note"), note, Text(".")), + Paragraph(Text("Half is "), new HEquation(fraction, default), Text(".")), + display, + Item(numbering, 0, "first bullet"), + Item(numbering, 1, "third numbered"), + Item(numbering, 2, "deep bullet"), + Item(numbering, 1, "fourth numbered"), + Item(numbering, 1, "its second paragraph", numbered: false), + Item(numbering, 0, "last bullet"), + new HParagraph { Format = new HParaFormat(Left: MarkupBlocks.QuoteIndent), Inlines = { Text("A quoted line.") } }, + Paragraph(Text("int main() {", MarkupBlocks.Code)), + Paragraph(Text(" return 0; // \\end{verbatim}? no", MarkupBlocks.Code)), + Paragraph(Text("}", MarkupBlocks.Code)), + table, + new HParagraph { HeadingLevel = 2, Inlines = { Text("Pictures") } }, + Paragraph(new HImage("C:/pictures/photo one.png")), + Paragraph(new HShape(HShapeKind.Line)), + Paragraph(Text("한글 텍스트 and café 😀"))); + document.Title = "Report & Notes"; + document.Authors.Add("Kim"); + return document; + } + + private static string? Pictures(string path) => "report_files/" + Uri.EscapeDataString(Path.GetFileName(path)); + + // ───────────────────────── LaTeX ───────────────────────── + + [Fact] + public void Latex_is_a_whole_article_with_everything_escaped() + { + var tex = LatexWriter.Write(Sample(), Pictures); + Assert.StartsWith("\\documentclass{article}\n", tex, StringComparison.Ordinal); + Assert.EndsWith("\\end{document}\n", tex, StringComparison.Ordinal); + foreach (var expected in new[] + { + // The preamble loads what the body uses. + "\\usepackage{iftex}", "\\usepackage{fontspec}", "\\usepackage{xeCJK}", "\\usepackage{amsmath,amssymb}", "\\usepackage{graphicx}", + "\\usepackage{longtable,array,multirow}", "\\usepackage[normalem]{ulem}", "\\usepackage{hyperref}", + "\\title{Report \\& Notes}", "\\author{Kim}", "\\maketitle", + "\\section{Intro\\_1}", "\\subsection{Pictures}", + "\\textbf{bold}", "\\emph{italic}", "\\protect\\uline{under}", "\\protect\\sout{gone}", "\\texttt{code()}", + "E=mc\\textsuperscript{2}", "H\\textsubscript{2}O", "Plain", "\\\\\nnext line", + // Specials + "a\\textbackslash{}b \\{c\\} \\$d \\& \\#e \\_f \\%g \\textasciitilde{}h \\textasciicircum{}i \\textless{}j\\textgreater{}", + "\\textbar{}k\\textbar{} \\textquotedbl{}l\\textquotedbl{} {[}m{]} n-{}-o '{}' \\textasciigrave{}p\\textasciigrave{}", + "\\href{https://example.com/a%20b\\#c}{the site}", "\\protect\\footnote{The note text.}", + "Half is $\\frac{a}{b}$.", "\\[\n\\sum_{i=1}^{n}i\n\\]", + "\\begin{itemize}\n\\item first bullet\n\\begin{enumerate}\n\\item[3.] third numbered\n\\begin{itemize}\n\\item deep bullet\n\\end{itemize}\n\\item[4.] fourth numbered\n\nits second paragraph\n\\end{enumerate}\n\\item last bullet\n\\end{itemize}", + "\\begin{quote}\nA quoted line.\n\n\\end{quote}", + "\\begin{longtable}", + "\\multicolumn{2}{|p{\\dimexpr 0.6667\\linewidth-2\\tabcolsep-\\arrayrulewidth\\relax}|}{wide}", "\\multirow{2}{=}{tall}", + "\\cline{1-2}", "\\endhead", "\\includegraphics{report_files/photo one.png}", "\\noindent\\rule{\\linewidth}{0.4pt}", + "한글 텍스트 and café 😀", + }) + Assert.True(tex.Contains(expected, StringComparison.Ordinal), $"lacks {expected}:\n{tex}"); + // Code holding \end{verbatim} can't be verbatim: its lines are typewriter text. + Assert.DoesNotContain("\\begin{verbatim}", tex, StringComparison.Ordinal); + Assert.Contains("{\\ttfamily int\\ main()\\ \\{\\\\", tex, StringComparison.Ordinal); + AssertBalancedLatex(tex); + } + + [Fact] + public void Latex_writes_code_verbatim_and_tex_math_as_its_source() + { + var read = MarkdownReader.Read("Text $\\alpha^2$ here.\n\n$$\\begin{align} a &= b \\end{align}$$\n\n```\nif (a < b) { c(); }\n```\n", fx.NewFolder()); + MarkupBlocks.OpenCodeBoxes(read); // as MarkupConverter does + var tex = LatexWriter.Write(read, _ => null); + Assert.Contains("Text $\\alpha^2$ here.", tex, StringComparison.Ordinal); + // A display environment of its own is written as it is, not inside \[ \]. + Assert.Contains("\\begin{align} a &= b \\end{align}", tex, StringComparison.Ordinal); + Assert.DoesNotContain("\\[\n\\begin{align}", tex, StringComparison.Ordinal); + Assert.Contains("\\begin{verbatim}\nif (a < b) { c(); }\n\\end{verbatim}", tex, StringComparison.Ordinal); + Assert.DoesNotContain("xeCJK", tex, StringComparison.Ordinal); + Assert.DoesNotContain("graphicx", tex, StringComparison.Ordinal); + AssertBalancedLatex(tex); + } + + [Fact] + public void Latex_lists_deeper_than_four_levels_stay_at_the_fourth() + { + var numbering = new HNumbering(); + numbering.Levels.Add(new HNumberingLevel("DIGIT", "•", 1, Bullet: true)); + var blocks = Enumerable.Range(0, 9).Select(level => (HBlock)Item(numbering, level, "level " + level)).ToArray(); + var tex = LatexWriter.Write(Document(blocks), _ => null); + Assert.Equal(4, Regex.Matches(tex, @"\\begin\{itemize\}").Count); + Assert.Contains("\\item level 8", tex, StringComparison.Ordinal); + AssertBalancedLatex(tex); + } + + /// + /// Braces balance (escaped ones and verbatim aside), environments close in order, and no [ follows \\ or \item + /// unbraced. + /// + private static void AssertBalancedLatex(string tex) + { + var environments = new Stack(); + var depth = 0; + for (var i = 0; i < tex.Length; i++) + { + if (string.CompareOrdinal(tex, i, "\\begin{verbatim}", 0, 16) == 0) + { + var end = tex.IndexOf("\\end{verbatim}", i + 16, StringComparison.Ordinal); + Assert.True(end > 0, "verbatim never ends"); + i = end + "\\end{verbatim}".Length - 1; + continue; + } + var m = Environment().Match(tex, i); + if (m.Success && m.Index == i) + { + if (m.Groups[1].Value == "begin") + environments.Push(m.Groups[2].Value); + else + Assert.Equal(environments.Pop(), m.Groups[2].Value); + } + switch (tex[i]) + { + case '\\': + i++; // an escaped character or the first letter of a control word + break; + case '{': + depth++; + break; + case '}': + depth--; + Assert.True(depth >= 0, $"unbalanced }} at {i}: {tex[Math.Max(0, i - 40)..Math.Min(tex.Length, i + 40)]}"); + break; + } + } + Assert.Equal(0, depth); + Assert.Empty(environments); + Assert.DoesNotMatch(@"(\\\\|\\item)\s*\[(?!\d+\.\]|\])", tex); + } + + [GeneratedRegex(@"\\(begin|end)\{([^}]*)\}")] + private static partial Regex Environment(); + + // ───────────────────────── AsciiDoc ───────────────────────── + + [Fact] + public void Asciidoc_writes_everything_and_keeps_text_from_reading_as_markup() + { + var adoc = AsciiDocWriter.Write(Sample(), Pictures); + Assert.StartsWith("= Report ++&++ Notes\n:stem: latexmath\n\n", adoc, StringComparison.Ordinal); + foreach (var expected in new[] + { + "== Intro++_++1", "=== Pictures", "**bold**", "__italic__", "[.underline]##under##", "[.line-through]##gone##", + "``code()``", "E=mc^2^", "H~2~O", " +\nnext line", + // Specials in passthroughs, + and | as attributes. + "a++\\++b ++{++c++}++ $d ++&++ ++#++e ++_++f %g ++~++h ++^++i ++<++j++>++ {vbar}k{vbar} \"l\" ++[++m++]++ n++-++-o", + "q++@++r ++*++s++*++ t{plus}u ++:++: v++;++; w++.++.. x++:++//y ++(++C) -? //z", + "link:++https://example.com/a%20b#c++[the site]", "footnote:[The note text.]", + "Half is latexmath:[\\frac{a}{b}].", "[latexmath]\n++++\n\\sum_{i=1}^{n}i\n++++", + "* first bullet\n.. third numbered\n*** deep bullet\n.. fourth numbered\n+\nits second paragraph\n* last bullet", + "____\nA quoted line.\n\n____", + "----\nint main() {\n return 0; // \\end{verbatim}? no\n}\n----", + "[cols=\"3*\",options=\"header\"]\n|===\n| H1\n| H2\n| H3\n\n2+| wide\n.2+| tall\n\n| x\n| y\n\n|===", + "image:report_files/photo%20one.png[]", "'''", + }) + Assert.True(adoc.Contains(expected, StringComparison.Ordinal), $"lacks {expected}:\n{adoc}"); + AssertNoBlockStarts(adoc); + } + + [Fact] + public void Asciidoc_guards_line_starts_directives_and_callouts() + { + var document = Document( + Paragraph(Text("* not a list"), new HLineBreak(default), Text("include::secret.txt[]"), new HLineBreak(default), Text(". not a title"), + new HLineBreak(default), Text("1. not an item"), new HLineBreak(default), Text("Term:: not a definition")), + Paragraph(Text("include::../../etc/passwd[]", MarkupBlocks.Code)), + Paragraph(Text("x = 1; <1>", MarkupBlocks.Code))); + var adoc = AsciiDocWriter.Write(document, _ => null); + Assert.Contains("{empty}++*++ not a list +\ninclude++:++:secret.txt++[]++ +\n{empty}. not a title +\n{empty}1. not an item +\nTerm++:++: not a definition", adoc, StringComparison.Ordinal); + Assert.Contains("[subs=\"specialchars\"]\n----\n\\include::../../etc/passwd[]\nx = 1; <1>\n----", adoc, StringComparison.Ordinal); + Assert.DoesNotMatch(new Regex("^include::", RegexOptions.Multiline), adoc); + } + + [Fact] + public void Asciidoc_numbers_a_list_from_its_first_number_and_nests_tables_once() + { + var numbering = new HNumbering(); + var level = new HNumberingLevel("DIGIT", "^1.", 5, Bullet: false); + numbering.Levels.Add(level); + var inner = new HTable { ColumnCount = 1 }; + var row = new HRow(); + row.Cells.Add(Cell("inner!cell")); + inner.Rows.Add(row); + var outer = new HTable { ColumnCount = 1 }; + var outerRow = new HRow(); + var cell = new HCell(); + cell.Blocks.Add(inner); + outerRow.Cells.Add(cell); + outer.Rows.Add(outerRow); + var adoc = AsciiDocWriter.Write(Document(Item(numbering, 0, "five"), Item(numbering, 0, "six"), outer), _ => null); + Assert.Contains("[start=5]\n. five\n. six", adoc, StringComparison.Ordinal); + Assert.Contains("a| [cols=\"1*\"]\n!===\n! inner\\!cell\n\n!===", adoc, StringComparison.Ordinal); + } + + /// Every line in a paragraph that isn't markup Filee wrote starts with a letter, digit or {empty}. + private static void AssertNoBlockStarts(string adoc) + { + foreach (var line in adoc.Split('\n')) + Assert.False(Regex.IsMatch(line, @"^(include|ifdef|ifndef|ifeval|endif)::"), line); + } + + // ───────────────────────── Typst ───────────────────────── + + [Fact] + public void Typst_writes_everything_and_escapes_its_markup() + { + var typ = TypstWriter.Write(Sample(), Pictures); + foreach (var expected in new[] + { + "#set document(title: \"Report & Notes\", author: (\"Kim\",))", "#align(center, text(size: 1.7em, weight: \"bold\")[Report & Notes])", + "= Intro\\_1", "== Pictures", "#strong[bold]", "#emph[italic]", "#underline[under]", "#strike[gone]", "`code()`", + "E=mc#super[2]", "H#sub[2]O", " \\\nnext line", + "a\\\\b {c} \\$d & \\#e \\_f %g \\~h ^i \\ |k| \\\"l\\\" \\[m\\] n\\--o \\'\\' \\`p\\` q\\@r \\*s\\* t+u :: v;; w\\... x\\:\\//y (C) \\-? \\//z", + "#link(\"https://example.com/a b#c\")[the site]", "#footnote[The note text.]", + "Half is $frac(a, b)$\\.", "$ ∑_(i = 1)^(n) i $", + "- first bullet\n 3. third numbered\n - deep bullet\n 4. fourth numbered\n\n its second paragraph\n- last bullet", + "#quote(block: true)[\nA quoted line.\n\n]", + "```\nint main() {\n return 0; // \\end{verbatim}? no\n}\n```", + "#table(\n columns: 3,\n table.header(\n [H1],\n [H2],\n [H3],\n ),\n table.cell(colspan: 2)[wide],\n table.cell(rowspan: 2)[tall],\n [x],\n [y],\n)", + "#image(\"report_files/photo one.png\")", "#line(length: 100%)", + }) + Assert.True(typ.Contains(expected, StringComparison.Ordinal), $"lacks {expected}:\n{typ}"); + AssertBalancedTypst(typ); + } + + [Fact] + public void Typst_guards_line_starts() + { + var typ = TypstWriter.Write(Document(Paragraph(Text("= not a heading"), new HLineBreak(default), Text("- not a list"), new HLineBreak(default), + Text("12. not an item"), new HLineBreak(default), Text("+ no"), new HLineBreak(default), Text("/ no: term"))), _ => null); + Assert.Contains("\\= not a heading \\\n\\- not a list \\\n12\\. not an item \\\n\\+ no \\\n\\/ no: term", typ, StringComparison.Ordinal); + } + + /// Square brackets and dollar signs pair up outside strings, raw text and escapes. + private static void AssertBalancedTypst(string typ) + { + var depth = 0; + var dollars = 0; + for (var i = 0; i < typ.Length; i++) + { + switch (typ[i]) + { + case '\\': + i++; + break; + case '"': + for (i++; i < typ.Length && typ[i] != '"'; i++) + if (typ[i] == '\\') + i++; + break; + case '`': + { + var fence = typ.Length - i >= 3 && typ.Substring(i, 3) == "```" ? "```" : "`"; + var end = typ.IndexOf(fence, i + fence.Length, StringComparison.Ordinal); + Assert.True(end > 0, "raw text never ends"); + i = end + fence.Length - 1; + break; + } + case '[': + depth++; + break; + case ']': + depth--; + Assert.True(depth >= 0, $"unbalanced ] at {i}"); + break; + case '$': + dollars++; + break; + } + } + Assert.Equal(0, depth); + Assert.Equal(0, dollars % 2); + } + + // ───────────────────────── man ───────────────────────── + + [Fact] + public void Man_page_writes_everything_in_ascii_requests() + { + var man = new ManWriter().Write(Sample()); + Assert.StartsWith(".TH \"Report & Notes\" \"1\" \"\" \"\" \"\"\n", man, StringComparison.Ordinal); + foreach (var expected in new[] + { + ".SH \"Intro_1\"", ".SS \"Pictures\"", "\\fBbold\\fP", "\\fIitalic\\fP", "\\fIunder\\fP", "\\f(CRcode()\\fP", "\n.br\nnext line", + "a\\eb {c} $d & #e _f %g \\(tih \\(hai |k| \\(dql\\(dq [m] n\\-\\-o \\(aq\\(aq \\(gap\\(ga q@r *s* t+u :: v;; w... x://y (C) \\-? //z", + "the site ", "and a note[1].", ".SH NOTES\n.IP \"[1]\" 5\nThe note text.", + "Half is a/b.", + ".IP \"\\(bu\" 4\nfirst bullet\n.RS 4\n.IP \"3.\" 4\nthird numbered\n.RS 4\n.IP \"\\(bu\" 4\ndeep bullet\n.RE\n.IP \"4.\" 4\nfourth numbered\n.IP \"\" 4\nits second paragraph\n.RE\n.IP \"\\(bu\" 4\nlast bullet", + ".RS 4\n.PP\nA quoted line.\n.RE", + ".EX\nint main() {\n\\& return 0; // \\eend{verbatim}? no\n}\n.EE", + ".TS\nallbox tab(@);\nlb lb lb\nl s l\nl l ^.\nT{\nH1\nT}@T{\nH2\nT}@T{\nH3\nT}\nT{\nwide\nT}@@T{\ntall\nT}\nT{\nx\nT}@T{\ny\nT}\n.TE", + "\\[uD55C]\\[uAE00] \\[uD14D]\\[uC2A4]\\[uD2B8] and caf\\[u00E9] \\[u1F600]", + }) + Assert.True(man.Contains(expected, StringComparison.Ordinal), $"lacks {expected}:\n{man}"); + Assert.DoesNotContain("photo", man, StringComparison.Ordinal); // a man page has no pictures + Assert.All(man, c => Assert.True(c < 0x80, $"U+{(int)c:X4}")); + // Every line starting with "." or "'" is a request Filee wrote. + foreach (var line in man.Split('\n').Where(l => l.StartsWith('.') || l.StartsWith('\''))) + Assert.Matches(@"^\.(TH|SH|SS|PP|br|IP|RS|RE|EX|EE|TS|TE|ce)( |$)", line); + Assert.Equal(Regex.Matches(man, "^\\.RS", RegexOptions.Multiline).Count, Regex.Matches(man, "^\\.RE", RegexOptions.Multiline).Count); + } + + [Fact] + public void Man_page_text_never_reads_as_a_request() + { + var man = new ManWriter().Write(Document(Paragraph(Text(".hidden request"), new HLineBreak(default), Text("'also hidden"), new HLineBreak(default), + Text(" indented"), new HLineBreak(default), Text("T} ends a cell")))); + Assert.Contains(".PP\n\\&.hidden request\n.br\n\\(aqalso hidden\n.br\n\\& indented\n.br\n\\&T} ends a cell", man, StringComparison.Ordinal); + } + + // ───────────────────────── Shared ───────────────────────── + + private static IEnumerable<(string Name, Func Write)> Writers(DocumentLimits l, CancellationToken cancellation) + { + yield return ("tex", d => new LatexWriter(_ => null) { Limits = l, Cancellation = cancellation }.Write(d)); + yield return ("adoc", d => new AsciiDocWriter(_ => null) { Limits = l, Cancellation = cancellation }.Write(d)); + yield return ("typ", d => new TypstWriter(_ => null) { Limits = l, Cancellation = cancellation }.Write(d)); + yield return ("man", d => new ManWriter { Limits = l, Cancellation = cancellation }.Write(d)); + } + + [Fact] + public void A_cancelled_conversion_stops_the_writers() + { + using var cancelled = new CancellationTokenSource(); + cancelled.Cancel(); + foreach (var (_, write) in Writers(DocumentLimits.Default, cancelled.Token)) + Assert.ThrowsAny(() => write(Sample())); + } + + [Fact] + public void A_footnote_shared_by_many_references_is_refused_by_the_limits() + { + var note = new HNote(endnote: false); + note.Blocks.Add(Paragraph(Text(new string('n', 100_000)))); + var paragraph = new HParagraph(); + for (var i = 0; i < 100; i++) + paragraph.Inlines.AddRange([Text("ref" + i + " "), note]); + var small = DocumentLimits.Default with { MaxPartBytes = 1024 * 1024 }; + foreach (var (name, write) in Writers(small, TestContext.Current.CancellationToken)) + { + var refusal = Assert.Throws(() => write(Document(paragraph))); + Assert.True(refusal.ErrorKey == DocumentLimits.ErrorKey, name); + } + } + + [Fact] + public void Hostile_lists_quotes_and_equations_are_written_in_time() + { + // List levels and quote indents far past what any language nests, and an equation 20,000 levels deep. + var numbering = new HNumbering(); + numbering.Levels.Add(new HNumberingLevel("DIGIT", "^1.", 1, Bullet: false)); + MathNode deep = new MathText("x"); + for (var i = 0; i < 20_000; i++) + deep = new MathRoot(null, deep); + var blocks = new List + { + Item(numbering, 10_000, "deep item"), + new HParagraph { Format = new HParaFormat(Left: int.MaxValue), Inlines = { Text("deep quote") } }, + Paragraph(new HEquation(deep, default, "deep equation")), + }; + var watch = System.Diagnostics.Stopwatch.StartNew(); + foreach (var (name, write) in Writers(DocumentLimits.Default, TestContext.Current.CancellationToken)) + { + var text = write(Document([.. blocks])); + foreach (var expected in new[] { "deep item", "deep quote", "deep equation" }) + Assert.True(text.Contains(expected, StringComparison.Ordinal), $"{name} lacks {expected}"); + } + Assert.True(watch.Elapsed < TimeSpan.FromSeconds(20), $"{watch.Elapsed}"); + } + + // ───────────────────────── Routes ───────────────────────── + + public static TheoryData Targets() => [.. MarkupConverter.WriteOnly.Keys]; + + [Theory] + [MemberData(nameof(Targets))] + public void Built_in_writers_win_over_pandoc_which_keeps_reading_them(string target) + { + foreach (var planner in new[] { fx.Catalog.CreatePlanner(["pandoc"]), MarkupTestKit.WithoutPandoc(fx).CreatePlanner() }) + { + foreach (var source in new[] { "md", "html", "docx", "odt", "hwpx", "hwp", "rtf", "rst", "org", "txt" }) + Assert.Equal("markup", Assert.Single(planner.Plan(source, target)!.Steps).Converter.Id); + } + var withPandoc = fx.Catalog.CreatePlanner(["pandoc"]); + Assert.Equal("pandoc", Assert.Single(withPandoc.Plan(target, "md")!.Steps).Converter.Id); + Assert.Null(MarkupTestKit.WithoutPandoc(fx).CreatePlanner().Plan(target, "md")); + } + + [Theory] + [MemberData(nameof(Targets))] + public async Task Markdown_converts_to_each_without_pandoc(string target) + { + var folder = fx.NewFolder(); + var source = Path.Combine(folder, "notes.md"); + await File.WriteAllTextAsync(source, "# Title_1\n\nSome *text* with $x^2$ and a [link](https://example.com).\n\n- one\n- two\n", + new UTF8Encoding(false), TestContext.Current.CancellationToken); + var output = await MarkupTestKit.ConvertAsync(MarkupTestKit.WithoutPandoc(fx), source, target); + var text = await File.ReadAllTextAsync(output, TestContext.Current.CancellationToken); + Assert.Contains("Title", text, StringComparison.Ordinal); + Assert.Contains("one", text, StringComparison.Ordinal); + Assert.Contains(target switch + { + "tex" => "$x^2$", + "adoc" => "latexmath:[x^2]", + "typ" => "$x^(2)$", + _ => "x\\(ha2", + }, text, StringComparison.Ordinal); + // Written through a temporary file, which is gone. + Assert.Empty(Directory.GetFiles(Path.GetDirectoryName(output)!, "*.tmp")); + } +} diff --git a/tests/Filee.Engines.Tests/MarkupRouteTests.cs b/tests/Filee.Engines.Tests/MarkupRouteTests.cs index 08999cc7c..dc67d83ce 100644 --- a/tests/Filee.Engines.Tests/MarkupRouteTests.cs +++ b/tests/Filee.Engines.Tests/MarkupRouteTests.cs @@ -28,10 +28,12 @@ public void Built_in_routes_win_over_pandoc_and_pandoc_keeps_the_rest(string mar Assert.Equal(engine, Assert.Single(planner.Plan(from, to)!.Steps).Converter.Id); } - // Slides come from the PPTX writer, which takes every built-in reader; what only Pandoc writes stays Pandoc's. + // Slides come from the PPTX writer, which takes every built-in reader; LaTeX is written here too (WriteOnly), + // and what only Pandoc reads stays Pandoc's. var withPandoc = fx.Catalog.CreatePlanner(["pandoc"]); Assert.Equal("pptx-writer", Assert.Single(withPandoc.Plan(markup, "pptx")!.Steps).Converter.Id); - Assert.Equal("pandoc", Assert.Single(withPandoc.Plan(markup, "tex")!.Steps).Converter.Id); + Assert.Equal("markup", Assert.Single(withPandoc.Plan(markup, "tex")!.Steps).Converter.Id); + Assert.Equal("pandoc", Assert.Single(withPandoc.Plan("tex", markup)!.Steps).Converter.Id); } [Fact] diff --git a/tests/Filee.Engines.Tests/MarkupWriterLimitTests.cs b/tests/Filee.Engines.Tests/MarkupWriterLimitTests.cs index 3e72c2671..52b9b29da 100644 --- a/tests/Filee.Engines.Tests/MarkupWriterLimitTests.cs +++ b/tests/Filee.Engines.Tests/MarkupWriterLimitTests.cs @@ -27,6 +27,10 @@ public class MarkupWriterLimitTests(EngineFixture fx) : IClassFixture MarkdownWriter.Write(d, _ => null)), ("ipynb", d => NotebookWriter.Write(d, fx.NewFolder(), pictures: false)), ("opml", d => OpmlWriter.Write(d, _ => null)), + ("tex", d => LatexWriter.Write(d, _ => null)), + ("adoc", d => AsciiDocWriter.Write(d, _ => null)), + ("typ", d => TypstWriter.Write(d, _ => null)), + ("man", d => new ManWriter().Write(d)), ]; private static T InTime(Func write) diff --git a/tests/Filee.Engines.Tests/PandocMarkupTests.cs b/tests/Filee.Engines.Tests/PandocMarkupTests.cs index 56162fdb6..dd3de807e 100644 --- a/tests/Filee.Engines.Tests/PandocMarkupTests.cs +++ b/tests/Filee.Engines.Tests/PandocMarkupTests.cs @@ -56,8 +56,11 @@ private async Task ConvertAsync(string input, string target) public void Markup_routes() { var planner = fx.Catalog.CreatePlanner(["pandoc"]); - foreach (var (from, to) in new[] { ("tex", "html"), ("tex", "epub"), ("docx", "tex"), ("md", "adoc") }) + foreach (var (from, to) in new[] { ("tex", "html"), ("tex", "epub"), ("tex", "adoc") }) Assert.Equal("pandoc", Assert.Single(planner.Plan(from, to)!.Steps).Converter.Id); + // LaTeX, AsciiDoc, Typst and man pages are written by Filee itself (read by Pandoc). + foreach (var (from, to) in new[] { ("docx", "tex"), ("md", "adoc"), ("html", "typ"), ("md", "man") }) + Assert.Equal("markup", Assert.Single(planner.Plan(from, to)!.Steps).Converter.Id); // reStructuredText is read and written by Filee itself. Assert.Equal("docx-writer", Assert.Single(planner.Plan("rst", "docx")!.Steps).Converter.Id); Assert.Equal("markup", Assert.Single(planner.Plan("rst", "md")!.Steps).Converter.Id); @@ -80,13 +83,14 @@ public void More_markup_languages_and_slides() foreach (var other in new[] { "md", "html", "docx", "odt" }) { // The languages Filee writes itself take the built-in writer. - Assert.Equal(Filee.Engines.Text.MarkupConverter.Writers.ContainsKey(markup) ? "markup" : "pandoc", + Assert.Equal(Filee.Engines.Text.MarkupConverter.Writers.ContainsKey(markup) || Filee.Engines.Text.MarkupConverter.WriteOnly.ContainsKey(markup) ? "markup" : "pandoc", Assert.Single(planner.Plan(other, markup)!.Steps).Converter.Id); Assert.NotNull(planner.Plan(markup, other)); } Assert.NotNull(planner.Plan(markup, "epub")); } - Assert.Equal("pandoc", Assert.Single(planner.Plan("org", "adoc")!.Steps).Converter.Id); + Assert.Equal("markup", Assert.Single(planner.Plan("org", "adoc")!.Steps).Converter.Id); + Assert.Equal("pandoc", Assert.Single(planner.Plan("typ", "adoc")!.Steps).Converter.Id); // Markdown and the markup languages Filee reads itself (Org) get slides from the built-in PPTX writer. Those it // reads only through Pandoc go through Pandoc's AST into the built-in writer when Pandoc is really there, else // (only assumed) through Pandoc's own PowerPoint writer. diff --git a/tests/Filee.Engines.Tests/TexMathWriterTests.cs b/tests/Filee.Engines.Tests/TexMathWriterTests.cs new file mode 100644 index 000000000..5b822e298 --- /dev/null +++ b/tests/Filee.Engines.Tests/TexMathWriterTests.cs @@ -0,0 +1,194 @@ +// The equation tree written as TeX (TexMathWriter: LaTeX, Markdown, AsciiDoc) and as Typst math (TypstMathWriter): +// TeX read into the tree, written back and read again gives the same tree; Word math (OMML) and 한글 equation scripts, +// which come as a tree, become TeX that Markdown reads back as the same equations; and a tree nested past the limit is +// left to the equation's text, quickly. + +using System.Diagnostics; +using System.Xml.Linq; +using Filee.Engines.Hwp.Hwpx; +using Filee.Engines.Hwp.Hwpx.Docx; + +namespace Filee.Engines.Tests; + +public class TexMathWriterTests(EngineFixture fx) : IClassFixture +{ + private static readonly XNamespace M = "http://schemas.openxmlformats.org/officeDocument/2006/math"; + + [Theory] + [InlineData(@"\frac{a+b}{2c}")] + [InlineData(@"x_i^2 + y^{n+1}")] + [InlineData(@"\sqrt[3]{x} + \sqrt{x^2+1}")] + [InlineData(@"\sum_{i=1}^{n} i^2")] + [InlineData(@"\int_0^\infty e^{-x}\,dx")] + [InlineData(@"\lim_{x \to 0} \frac{\sin x}{x}")] + [InlineData(@"\begin{pmatrix} a & b \\ c & d \end{pmatrix}")] + [InlineData(@"\begin{vmatrix} a & b \\ c & d \end{vmatrix} + \begin{bmatrix} 1 \\ 2 \end{bmatrix}")] + [InlineData(@"|x| = \begin{cases} x & x \ge 0 \\ -x & \text{otherwise} \end{cases}")] + [InlineData(@"\left( \frac{a}{b} \right) + \left\{ x \right. + \left\langle y \right\rangle")] + [InlineData(@"\hat{x} + \overline{AB} + \vec{v} + \tilde{y} + \underline{z}")] + [InlineData(@"\mathbf{v} \cdot \mathrm{d}x + \boldsymbol{\alpha}")] + [InlineData(@"\alpha \leq \beta \neq \gamma \to \infty")] + [InlineData(@"\binom{n}{k}")] + [InlineData(@"\begin{aligned} a &= b \\ c &= d \end{aligned}")] + [InlineData(@"f'(x) + f''(y)")] + [InlineData(@"\{x\} \cup \emptyset")] + [InlineData(@"x \in \mathbb{R} \subseteq \mathcal{L}")] + [InlineData(@"\operatorname{lcm}(a, b) + \max_{i} x_i")] + [InlineData(@"a \quad b \qquad c \, d \; e")] + [InlineData(@"\not\in \not= \pm \times \div")] + [InlineData(@"\Gamma \Delta \Omega \omega")] + public void Tex_written_from_the_tree_reads_back_as_the_same_tree(string tex) + { + var tree = LatexMath.Parse(tex)!; + var written = TexMathWriter.Write(tree); + Assert.NotNull(written); + var again = LatexMath.Parse(written); + Assert.NotNull(again); + Assert.Equal(EquationScript.Script(tree), EquationScript.Script(again)); + } + + [Fact] + public void Characters_tex_treats_specially_are_escaped() + { + var tree = new MathRow([ + new MathQuoted("50% & #1 {a} $x_y ~^\\"), + new MathText("한"), + new MathCommand("Alpha", "Α"), + new MathSymbol("{"), + new MathSymbol("#"), + ]); + var tex = TexMathWriter.Write(tree)!; + Assert.Contains(@"\text{50\% \& \#1 \{a\} \$x\_y \textasciitilde{}\textasciicircum{}\textbackslash{}}", tex, StringComparison.Ordinal); + Assert.Contains(@"\text{한}", tex, StringComparison.Ordinal); + Assert.Contains("A", tex, StringComparison.Ordinal); + Assert.DoesNotContain("Α", tex, StringComparison.Ordinal); // Greek Alpha has no command: the Latin letter + Assert.Contains(@"\{\#", tex, StringComparison.Ordinal); + } + + [Fact] + public void A_control_word_is_kept_apart_from_a_letter_after_it() + { + var tex = TexMathWriter.Write(new MathRow([new MathCommand("alpha", "α"), new MathText("x"), new MathFunction("sin"), new MathText("y")]))!; + Assert.Equal(@"\alpha x\sin y", tex); + } + + [Fact] + public void Word_math_becomes_tex() + { + // (a + b) / 2 under a square root, with x² beside it. + var oMath = new XElement(M + "oMath", + new XElement(M + "rad", new XElement(M + "radPr", new XElement(M + "degHide", new XAttribute(M + "val", "1"))), new XElement(M + "deg"), + new XElement(M + "e", + new XElement(M + "f", + new XElement(M + "num", Run("a+b")), + new XElement(M + "den", Run("2"))))), + new XElement(M + "sSup", new XElement(M + "e", Run("x")), new XElement(M + "sup", Run("2")))); + var tex = TexMathWriter.Write(OmmlReader.Read(oMath))!; + Assert.Equal(@"\sqrt{\frac{a+b}{2}}x^{2}", tex); + + static XElement Run(string text) => new(M + "r", new XElement(M + "t", text)); + } + + [Fact] + public void A_hangul_equation_becomes_tex() + { + var tex = TexMathWriter.Write(EquationParser.Parse("{a} over {b} + sqrt {x} + SUM _{i=1} ^{n} i + alpha LEQ beta"))!; + Assert.Equal(@"\frac{a}{b}+\sqrt{x}+\sum_{i=1}^{n}i+\alpha\leq\beta", tex); + } + + [Theory] + [InlineData(@"\frac{a}{b}", "frac(a, b)")] + [InlineData(@"x_i^2", "x_(i)^(2)")] + [InlineData(@"\sqrt[3]{x} + \sqrt{y}", "root(3, x) + sqrt(y)")] + [InlineData(@"\sum_{i=1}^{n} i", "∑_(i = 1)^(n) i")] + [InlineData(@"\lim_{x \to 0} \sin x", "lim_(x → 0) op(\"sin\") x")] + [InlineData(@"\begin{pmatrix} a & b \\ c & d \end{pmatrix}", "mat(delim: \"(\", a, b; c, d)")] + [InlineData(@"\begin{cases} 1 & x > 0 \\ 0 & \text{else} \end{cases}", "cases(1 & x > 0, 0 & \"else\")")] + [InlineData(@"\left| x \right| + \left( a \right)", "abs(x) + (a)")] + [InlineData(@"\hat{x} + \vec{v} + \overline{AB}", "hat(x) + arrow(v) + overline(A B)")] + [InlineData(@"\mathbf{v} + \mathrm{d}", "bold(upright(v)) + upright(d)")] + [InlineData(@"\binom{n}{k}", "binom(n, k)")] + [InlineData(@"f'(x)", "f' \\( x \\)")] + [InlineData(@"ab", "a b")] + [InlineData(@"\frac{f(a,b)}{2}", "frac(f \\( a \\, b \\), 2)")] + [InlineData(@"a \quad b \, c", "a quad b thin c")] + public void Tex_becomes_typst_math(string tex, string typst) => + Assert.Equal(typst, TypstMathWriter.Write(LatexMath.Parse(tex)!)); + + [Fact] + public void Aligned_lines_are_typst_lines_with_their_marks() + { + var typst = TypstMathWriter.Write(LatexMath.Parse(@"\begin{aligned} a &= b \\ c &= d \end{aligned}")!); + Assert.Equal("a & = b \\\nc & = d", typst); + // Inside something else, lines are a matrix. + Assert.StartsWith("(mat(delim: #none", TypstMathWriter.Write(LatexMath.Parse(@"\left( \begin{gathered} a \\ b \end{gathered} \right)")!), StringComparison.Ordinal); + } + + [Fact] + public void Typst_escapes_its_syntax_and_quotes_text() + { + var typst = TypstMathWriter.Write(new MathRow([new MathQuoted("say \"hi\" \\ now"), new MathText("한글"), new MathSymbol("#"), new MathSymbol("$"), new MathSymbol("/")]))!; + Assert.Equal("\"say \\\"hi\\\" \\\\ now\" \"한글\" \\# \\$ \\/", typst); + } + + // ───────────────────────── Markdown ───────────────────────── + + private static HDocument Document(params HParagraph[] paragraphs) + { + var document = new HDocument(); + var section = new HSection(); + section.Blocks.AddRange(paragraphs); + document.Sections.Add(section); + return document; + } + + [Fact] + public void Markdown_writes_word_and_hangul_equations_as_tex_that_it_reads_back() + { + var inline = new HParagraph(); + var fraction = new MathFraction(new MathRow([new MathText("a"), new MathSymbol("+"), new MathText("b")]), new MathNumber("2")); + inline.Inlines.AddRange([new HText("Half: ", default), new HEquation(fraction, default), new HText(" done.", default)]); + // Word's display equation: alone in a centred paragraph. + var display = new HParagraph { Format = new HParaFormat(Align: HAlign.Center) }; + var sum = EquationParser.Parse("SUM _{i=1} ^{n} i ^{2}"); + display.Inlines.Add(new HEquation(sum, default)); + + var markdown = MarkdownWriter.Write(Document(inline, display), _ => null); + Assert.Contains(@"Half: $\frac{a+b}{2}$ done.", markdown, StringComparison.Ordinal); + Assert.Contains(@"$$\sum_{i=1}^{n}i^{2}$$", markdown, StringComparison.Ordinal); + Assert.DoesNotContain("(a+b)/2", markdown, StringComparison.Ordinal); + + var read = MarkdownReader.Read(markdown, fx.NewFolder()); + var equations = HDocumentWalker.Inlines(read.Sections.SelectMany(s => s.Blocks)).OfType().ToList(); + Assert.Equal(2, equations.Count); + Assert.Equal(EquationScript.Script(fraction), EquationScript.Script(equations[0].Math)); + Assert.Equal(EquationScript.Script(sum), EquationScript.Script(equations[1].Math)); + var paragraphs = read.Sections.SelectMany(s => s.Blocks).OfType().ToList(); + Assert.Equal(HAlign.Center, paragraphs.Single(p => p.Inlines.Contains(equations[1])).Format.Align); + } + + [Fact] + public void Markdown_keeps_tex_it_read_as_written() + { + var read = MarkdownReader.Read("Area $\\pi r^2$ here.\n", fx.NewFolder()); + Assert.Contains("$\\pi r^2$", MarkdownWriter.Write(read, _ => null), StringComparison.Ordinal); + } + + // ───────────────────────── Depth ───────────────────────── + + [Fact] + public void A_tree_nested_thousands_deep_is_left_to_its_text_in_time() + { + MathNode node = new MathText("x"); + for (var i = 0; i < 20_000; i++) + node = new MathFraction(node, new MathNumber("2")); + var watch = Stopwatch.StartNew(); + Assert.Null(TexMathWriter.Write(node)); + Assert.Null(TypstMathWriter.Write(node)); + // The writers then write the equation's text (here its given source). + var paragraph = new HParagraph(); + paragraph.Inlines.Add(new HEquation(node, default, "deep equation")); + Assert.Contains("deep equation", MarkdownWriter.Write(Document(paragraph), _ => null), StringComparison.Ordinal); + Assert.True(watch.Elapsed < TimeSpan.FromSeconds(20), $"{watch.Elapsed}"); + } +} From fd6010bde6406cafaa6b88146ec6d259cb9172bc Mon Sep 17 00:00:00 2001 From: KnifeLemon Date: Fri, 20 Nov 2026 21:15:39 +0900 Subject: [PATCH 3/6] Write UYAP documents (.udf) with a built-in writer (#1546, #1633) Anything Filee reads (Word, ODT, RTF, HWP/HWPX, Markdown, HTML, PDF, e-books, ...) now converts to UDF, the UYAP Document Editor's file, in one step through the new udf-writer engine. The writer works from the document model like the RTF and ODT writers and writes content.xml the way the editor does: one CDATA text with the header, body and footer, runs and pictures by offset, the page format, paragraphs with alignment, indents, spacing, line spacing, tab stops and lists, runs with font, size, bold/italic/underline/strike and colours, tables with column widths, colspan, fill, alignment and cell borders, pictures as PNG or JPEG, page breaks, one header and one footer. - Text XML can't hold is left out without moving the offsets; "]]>" splits the CDATA block. - content.xml stays within what the reader takes (part size, elements, paragraphs, objects, inflate ratio); pictures past the limits are left out; a refusal, failure or cancellation leaves no file. - UDF becomes writable. The reader now reads lists (Numbered/Bulleted, ListId, ListLevel), header rows, row heights and the cells' align, fillColor, borderNone and borderSpec. - UDF tables go to XLSX, ODS and CSV through Document tables; pictures of the pages come from the Filee renderer's PDF. - UDF -> folder (Extract) is taken only from a .udf itself, not from a document written as UDF on the way. Co-Authored-By: Claude Opus 5.5 --- docs/ENGINES.md | 76 +- src/Filee.App/Assets/i18n/ar.json | 2 + src/Filee.App/Assets/i18n/bn-BD.json | 2 + src/Filee.App/Assets/i18n/de.json | 2 + src/Filee.App/Assets/i18n/en.json | 2 + src/Filee.App/Assets/i18n/es.json | 2 + src/Filee.App/Assets/i18n/fa.json | 2 + src/Filee.App/Assets/i18n/fr.json | 2 + src/Filee.App/Assets/i18n/hi.json | 2 + src/Filee.App/Assets/i18n/id.json | 2 + src/Filee.App/Assets/i18n/it.json | 2 + src/Filee.App/Assets/i18n/ja.json | 2 + src/Filee.App/Assets/i18n/ko.json | 2 + src/Filee.App/Assets/i18n/mn.json | 2 + src/Filee.App/Assets/i18n/ne.json | 2 + src/Filee.App/Assets/i18n/pl.json | 2 + src/Filee.App/Assets/i18n/pt-BR.json | 2 + src/Filee.App/Assets/i18n/pt-PT.json | 2 + src/Filee.App/Assets/i18n/ru.json | 2 + src/Filee.App/Assets/i18n/th.json | 2 + src/Filee.App/Assets/i18n/tr.json | 2 + src/Filee.App/Assets/i18n/vi.json | 2 + src/Filee.App/Assets/i18n/zh-CN.json | 2 + src/Filee.App/Assets/i18n/zh-TW.json | 2 + src/Filee.Core/Formats/FormatRegistry.cs | 2 +- .../Archives/ArchiveConverter.cs | 5 +- src/Filee.Engines/EngineRegistry.cs | 1 + .../Office/DocumentTablesConverter.cs | 12 +- src/Filee.Engines/Uyap/UyapReader.cs | 130 ++- src/Filee.Engines/Uyap/UyapWriter.cs | 833 ++++++++++++++++++ src/Filee.Engines/Uyap/UyapWriterConverter.cs | 45 + tests/Filee.Engines.Tests/UyapTests.cs | 45 +- tests/Filee.Engines.Tests/UyapWriterTests.cs | 503 +++++++++++ 33 files changed, 1669 insertions(+), 29 deletions(-) create mode 100644 src/Filee.Engines/Uyap/UyapWriter.cs create mode 100644 src/Filee.Engines/Uyap/UyapWriterConverter.cs create mode 100644 tests/Filee.Engines.Tests/UyapWriterTests.cs diff --git a/docs/ENGINES.md b/docs/ENGINES.md index 3ed3dc706..a1642ed42 100644 --- a/docs/ENGINES.md +++ b/docs/ENGINES.md @@ -22,6 +22,7 @@ Filee chooses engines automatically. Settings → *Engines* shows their status, | **HWPX writer** | built in | **HWP, DOCX (+ DOCM/DOTX/DOTM), DOC (+ DOT, WPS Office WPS), ODT (+ OTT, FODT), XLSX (+ XLSM/XLTX), XLS, ODS (+ OTS, FODS), CSV, TSV, PPTX (+ PPTM/POTX/POTM/PPSX), PPT, ODP (+ OTP, FODP), PDF, OFD, UYAP (UDF), HTML, MHTML, EPUB, MOBI/AZW3, FB2, CHM, PDB, PML, TCR, RB, SNB, OEB, HWPX, TXT, Markdown, RTF, mind maps (MM, XMind) → HWPX** (and with rhwp → PDF and images); reStructuredText, LaTeX → HWPX with Pandoc | | **DOCX writer** | built in | **DOC (+ DOT, WPS Office WPS), ODT (+ OTT, FODT), HWPX, HWP, PDF, OFD, UYAP (UDF), HTML, MHTML, EPUB, MOBI/AZW3, FB2 (and the older e-books), Markdown, TXT, RTF, XLSX, XLS, ODS (+ OTS, FODS), CSV, PPTX (+ variants), PPT, ODP (+ OTP, FODP), mind maps (MM, XMind) → DOCX** | | **RTF writer** | built in | **DOCX (+ variants), DOC (+ DOT, WPS), ODT, HWPX, HWP, PDF, OFD, UYAP (UDF), HTML, MHTML, EPUB, MOBI/AZW3, FB2 (and the older e-books), Markdown, TXT, XLSX, XLS, ODS, CSV, PPTX (+ variants), PPT, ODP (+ OTP, FODP), mind maps (MM, XMind) → RTF** (Rich Text Format, see [RTF writer](#rtf-writer)) | +| **UYAP writer** | built in | **DOCX (+ variants), DOC (+ DOT, WPS), ODT (+ OTT, FODT), RTF, HWPX, HWP, PDF, OFD, HTML, MHTML, EPUB, MOBI/AZW3, FB2 (and the older e-books), Markdown, TXT, XLSX, XLS, ODS, CSV, PPTX (+ variants), PPT, ODP (+ OTP, FODP), mind maps (MM, XMind) → UDF** (the UYAP Document Editor's file, see [UYAP writer](#uyap-writer)) | | **ODT writer** | built in | **DOCX (+ variants), DOC (+ DOT, WPS), HWPX, HWP, PDF, OFD, UYAP (UDF), HTML, MHTML, EPUB, MOBI/AZW3, FB2 (and the older e-books), Markdown, TXT, RTF, XLSX, XLS, ODS (+ OTS, FODS), CSV, PPTX (+ variants), PPT, ODP (+ OTP, FODP), mind maps (MM, XMind) → ODT** (OpenDocument text, LibreOffice Writer) | | **PPTX writer** | built in | **Markdown, DOCX (+ variants), DOC (+ DOT, WPS), ODT (+ OTT, FODT), HWPX, HWP, OFD, HTML, MHTML, EPUB, MOBI/AZW3, FB2 (and the older e-books), TXT, RTF, XLSX, XLS, ODS (+ OTS, FODS), CSV → PPTX slides** (a slide per heading, Pandoc's rules; Pandoc's markup languages too when Pandoc is installed); **PPT → PPTX** as the [PPT reader](#ppt-reader) translates it | | **ODP writer** | built in | **PPTX (+ PPTM/POTX/POTM/PPSX), PPT → ODP, OTP, FODP** with masters, shapes, pictures, tables, lists and speaker notes; **Markdown, DOCX (+ variants), DOC (+ DOT, WPS), ODT (+ OTT, FODT), HWPX, HWP, OFD, HTML, MHTML, EPUB, MOBI/AZW3, FB2 (and the older e-books), TXT, RTF, XLSX, XLS, ODS, CSV → ODP slides** laid out as the PPTX writer lays them out (see [ODP writer](#odp-writer)); a deck with what the slide model leaves out (charts without a picture, effects, 3D) keeps LibreOffice's route | @@ -29,7 +30,7 @@ Filee chooses engines automatically. Settings → *Engines* shows their status, | PDF slides | built in (PdfPig, PDFium) | **PDF → PPTX**: each page rebuilt as an editable slide (text boxes with the PDF's fonts, pictures, shapes, ruled tables); or, as a preset option, a picture of each page with its text in the speaker notes | | PDF tables | built in (PdfPig) | **PDF → XLSX, ODS, CSV** (#267): each table drawn with lines on its own sheet, numbers and dates as real values, merged cells and column widths kept | | **PostScript writer** | built in | **PDF → PS (LanguageLevel 3), EPS** (one per page) without Ghostscript: vectors, text with its embedded fonts, pictures, shadings, patterns; pages with transparency as pictures (see [PDF → PostScript and EPS](#pdf--postscript-and-eps)) | -| Document tables | built in | **DOCX (+ DOCM/DOTX/DOTM), DOC (+ DOT/WPS), ODT (+ OTT, FODT), RTF, HWP, HWPX, HTML, MHTML → XLSX, ODS, CSV** (#1016): each table on its own sheet, numbers and dates as real values, merged cells, borders, fill, alignment and column widths kept | +| Document tables | built in | **DOCX (+ DOCM/DOTX/DOTM), DOC (+ DOT/WPS), ODT (+ OTT, FODT), RTF, HWP, HWPX, HTML, MHTML, UYAP (UDF) → XLSX, ODS, CSV** (#1016, #1588): each table on its own sheet, numbers and dates as real values, merged cells, borders, fill, alignment and column widths kept | | E-mail | built in (MimeKit, MsgReader) | **EML, Outlook MSG → HTML** (headers + body, inline pictures), **TXT**, **ZIP** (the attachments); **MSG → EML**; **MHTML → HTML** (the saved page in one file, pictures inlined; on to PDF, DOCX, HWPX, TXT, Markdown, EPUB from there) | | Signed files | built in (System.Security.Cryptography.Pkcs) | **P7M → the signed file inside** (CMS / PKCS #7, CAdES: Italian e-invoices `*.xml.p7m`, signed PDFs `*.pdf.p7m`), then on from its format (P7M → PDF → JPG); the signature isn't checked | | Data | built in (System.Text.Json, YamlDotNet, Tomlyn, System.Xml) | **JSON, JSON Lines, YAML, TOML, XML, CSV ↔ each other** (CSV for tables); **vCard, iCalendar ↔ CSV**; **property lists (.plist, XML and binary) → each of them and TXT**; FreeMASTER oscilloscope TXT → CSV; spreadsheets, contacts and events reach JSON / YAML / TOML / XML through CSV | @@ -2315,9 +2316,10 @@ full 140 mm invoice opened there puts its last line on a second page (rhwp's PDF UYAP's `.udf` (#1338) is the file of the UYAP Document Editor (UYAP Doküman Editörü), which Turkish courts, lawyers and citizens write petitions and decisions in. `Uyap/UyapReader.cs` reads it into the document model with no external engine, so UDF → PDF is one step through the Filee renderer (two, HWPX writer and rhwp, with it off), UDF → DOCX, ODT -and HWPX one through their writers, and TXT, Markdown, HTML and EPUB follow from there. UDF is read only -(`Writable: false`) and sits on the office donut (new libraries; an existing library learns `.udf` through a settings -migration). +and HWPX one through their writers, and TXT, Markdown, HTML and EPUB follow from there. Pictures of the pages (PNG, +JPEG, #1633) are drawn from that PDF as a Word document's are, and the tables go to XLSX, ODS and CSV through Document +tables. UDF is written too ([UYAP writer](#uyap-writer)) and sits on the office donut (new libraries; an existing +library learns `.udf` through a settings migration). - **Package**: a ZIP with one part, `content.xml`, read within `DocumentLimits` (`Documents/DocumentPackage`: part sizes counted while unpacking, the inflate ratio, no DTD, depth and element limits). Its `