From 2e614c2002830ea22608b970d5add990e6b2b3ab Mon Sep 17 00:00:00 2001 From: Evan Gress <106449014+evangress@users.noreply.github.com> Date: Mon, 9 Nov 2026 21:43:26 -0500 Subject: [PATCH 1/4] fix(serializer): stop escaping Obsidian syntax and plain prose on save MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit remark-stringify escapes every `[`, `*`, `_` and `~` defensively. For an Obsidian vault that is destructive: saving an edited note rewrote `[[Note]]` as `\[\[Note]]`, `![[img.png]]` as `!\[\[img.png]]` and `> [!tip]` as `> \[!tip]`, which Obsidian renders as literal text. It also rewrote ordinary prose (`figure [1]`, `~3 cm`, `snake_case`). src/editor/escapes.ts removes each escape only when re-parsing proves the document unchanged: per-block bisection (memoized), then one whole-document parse as the proof, with a bounded whole-document fallback. It never re-implements CommonMark's escaping rules. Gates: tests/escapes.test.ts (search, fallback, parse budget, 200-case property over the real parser) and new obsidianPreserved / escapesKept classes plus a repair fixture in roundtrip.test.ts. Also adds Track H (hardening), Track M (medical & radiology students) and §9 (unplaced value areas) to ROADMAP.md, and fixes a stale README line. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 19 +++ CLAUDE.md | 39 +++++- README.md | 6 +- ROADMAP.md | 267 ++++++++++++++++++++++++++++++++++++++- src/editor/escapes.ts | 235 ++++++++++++++++++++++++++++++++++ src/editor/serializer.ts | 44 ++++++- src/welcome.ts | 2 +- tests/escapes.test.ts | 189 +++++++++++++++++++++++++++ tests/roundtrip.test.ts | 79 +++++++++++- 9 files changed, 865 insertions(+), 15 deletions(-) create mode 100644 src/editor/escapes.ts create mode 100644 tests/escapes.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index a58575f..83f1103 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,25 @@ GitHub Release notes plus the commits that shipped in it. ## [Unreleased] +### Fixed +- **Saving a note no longer breaks Obsidian links, embeds and callouts.** When you edited a + note in an Obsidian vault, saving it rewrote `[[Pneumothorax]]` as `\[\[Pneumothorax]]`, + `![[scan.png]]` as `!\[\[scan.png]]`, and `> [!tip]` as `> \[!tip]`. Obsidian shows those + as plain text, so every link, embedded image and callout in that note stopped working. + Notes you only opened were never affected; notes you edited were. They now save + exactly as written. + + If a note was saved by an earlier version, its links are still escaped on disk. Removing + the backslashes is safe: open the note in Toril, make any edit, and save. Toril writes + the links back without them. Version history also holds the copy from before Toril + first saved it. + +- **Toril no longer adds backslashes to ordinary text.** `figure [1]`, `~3 cm`, `5 * 3`, + `snake_case` and bare web addresses are left as you typed them. A backslash is now + written only where the text would otherwise change meaning, such as a line that would + turn into a heading or list. Every save checks this by re-reading what it is about to + write and comparing it with your note, so no backslash is removed where it is needed. + ## [v1.1.1] — 2026-10-15 ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index bd35e1e..09d8f07 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -54,13 +54,32 @@ the feature-by-feature record. **Known trade-off:** Toril's canonical form (`src/editor/canonical.ts`) is `-` bullets and `---` thematic breaks, matching Obsidian — so most notes survive open→save untouched. Some constructs still reformat (setext headings, indented code blocks, `~~~` fences, two-space hard breaks, link -reference definitions, `*`/`+`-authored bullets, `1)` ordered markers, over-indented nesting, bare -URLs, and **CRLF line endings → LF**, which rewrites every line of a Windows-authored note); the +reference definitions, `*`/`+`-authored bullets, `1)` ordered markers, over-indented nesting, +`` autolinks → bare URLs, superfluous backslash escapes dropped, and **CRLF line +endings → LF**, which rewrites every line of a Windows-authored note); the `normalized` class in `tests/roundtrip.test.ts` is the authoritative list — each entry pinned to its exact output. The reformatting is idempotent and never drops *rendered* content; the one known exception is that an **unused link reference definition is deleted outright** (see the deferred list above). Relevant to Obsidian-vault diffs (§1). +**Escapes are minimal and proven — `fix/obsidian-syntax-escaping`.** remark-stringify escapes +defensively (every `[`, `*`, `_`, `~`), which was a §3 bug, not a style issue: saving an edited +note rewrote `[[Note]]` as `\[\[Note]]`, `![[x.png]]` as `!\[\[x.png]]` and `> [!tip]` as +`> \[!tip]` — literal text in Obsidian, so every wikilink, image embed and callout in that note +broke. It also rewrote ordinary prose (`figure [1]`, `~3 cm`, `snake_case`). `docToMarkdown` +(`serializer.ts`) now passes the escaped output through `src/editor/escapes.ts`, which removes +each escape **only if re-parsing proves the document unchanged** — never by re-implementing +CommonMark's escaping rules. Per-block bisection (memoized, so between keystrokes only the edited +block re-parses), then one whole-document parse as the proof; if that fails (a block whose meaning +depends on another, e.g. a reference definition), a bounded whole-document search. Cost: roughly +one extra full parse per serialization (~12 ms at 800 words, ~180 ms at 12k words in jsdom), paid +on the debounced change listener. Rules for whoever touches it: the property gate compares against +the **fully escaped** output, not the live doc, because two upstream quirks are lossy or unstable +with or without this module — remark moves a space out of a strikethrough boundary +(`~~a ~~b` shapes), and **a bare URL ending in `\` gains a backslash on every save** (pre-existing on +`main`, not yet fixed). Table column padding is computed before minimization, so a cell holding +`[[x\|y]]` may pad a character or two wide (pinned as `wikilinkAliasInTable`). + **HTML as a first-class editable format — shipped in v0.1.1-alpha.1.** Open/edit/save `.html` WYSIWYG alongside `.md`, motivated by AI assistants emitting rich HTML instead of Markdown. The editor engine is unchanged — HTML is the native shape of a ProseMirror doc, so it is a second canonical @@ -344,6 +363,7 @@ toril/ │ ├── editor/ │ │ ├── milkdown.ts # WYSIWYG setup + plugins (incl. clipboard-image paste, §6) │ │ ├── serializer.ts # the ONE markdown <-> doc converter (§3.2) +│ │ ├── escapes.ts # proof-checked escape minimization, used only by serializer.ts │ │ ├── html-serializer.ts # the ONE HTML <-> doc converter (§3.2/§3.3) │ │ └── html-constructs.ts # richer HTML-only schema: callout/details/dl/mark/sub/sup (§6) │ ├── ui/ @@ -657,15 +677,24 @@ Phases 0–3 are complete and Phase 4 (polish) is in progress; the shipped detai **Gates (all green) — keep them green:** - **Atomic save:** `cargo test -p fsatomic` — interrupting a save leaves the original intact (§3.1). - **Round-trip:** `tests/roundtrip.test.ts` — real Milkdown in jsdom, built through - `canonical.ts` so the gate tests the canon that ships. Three classes: `fixtures` + `canonical.ts` so the gate tests the canon that ships. Classes: `fixtures` (canonical input is stable), `preserved` (human/Obsidian-authored input is **not** - rewritten), `normalized` (what we do rewrite, pinned to exact output), and + rewritten), `obsidianPreserved` (wikilinks, embeds, callouts, `==highlight==`, + `%%comments%%`, block IDs, nested tags, Anki cloze — Obsidian syntax Toril does not + render yet but must never break), `escapesKept` (the escapes that *are* needed — + task markers, heading/list/quote markers, raw HTML — stay), `normalized` (what we + do rewrite, pinned to exact output), and `frontMatter` (**whole-file** trip — split, round-trip the body, rejoin — the class that pins the §3 front-matter fix, including blocks no serializer could reproduce). Add math fixtures if math ever lands (§3.2). Plus `tests/frontmatter.test.ts`, the splitter's own gate: `join(split(x)) === x` over fixtures *and* 2000 randomized documents, detection per format, and the Rust-parity cases mirrored in - `mdhtml`/`mdrtf`. + `mdhtml`/`mdrtf`. Plus `tests/escapes.test.ts` — the minimizer's own gate: the + bisection (joint verification, so `\*a\*` never loses both escapes), the + whole-document fallback, the parse budget, a throwing parser read as *inequality*, + and a 200-case randomized property over the real parser (the saved file loads + exactly as the fully escaped file would, and is stable wherever the old output was). + Verified with teeth: forcing the verifier to accept everything fails 8 tests. - **Front-matter properties:** `tests/frontmatter-values.test.ts` — Obsidian's own writes come back as typed rows (the feature is useless otherwise), and everything we would *reformat* lands in raw mode instead: comments, anchors, flow sequences, diff --git a/README.md b/README.md index 4aa8624..464445d 100644 --- a/README.md +++ b/README.md @@ -56,8 +56,10 @@ distinction matters, so they are listed separately. **Not yet built:** -- Math rendering (KaTeX), PDF export, and in-editor YAML front matter. -- Global search across a vault, wikilinks, and backlinks. +- Math rendering (KaTeX) and PDF export. +- Global search across a vault, and wikilink navigation and backlinks. + Obsidian's `[[links]]`, `![[embeds]]` and `> [!callouts]` are kept intact when + Toril saves a note, but are not yet clickable or rendered. - Signed installers, so Windows SmartScreen and macOS Gatekeeper warn on first run. That is expected for an unsigned build, not a sign of a problem. diff --git a/ROADMAP.md b/ROADMAP.md index 9ef1c6e..8790b33 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -96,6 +96,39 @@ no AI. That gap is this roadmap. > gospel, and the tag has run ahead of it twice now; §3's trust-before-reach rule still > governs what may be *reached* from the UI, regardless of version. +> **Status update (2026-11-09) — who is actually using it, and a §3 fix it surfaced.** +> Toril is seeing heavy use by **radiology / medical students**. That audience changes +> priorities: their notes are image-heavy, structured around cases, studied rather than +> just written, often live in an Obsidian vault they also read on a phone, and sometimes +> brush against patient-identifying images. Two new tracks below — **Track H +> (hardening)** and **Track M (medical & radiology students)** — and §9 (value Toril +> does not offer today) come out of that review. +> +> Reviewing for that audience found a **§3 round-trip bug**: saving an *edited* note +> backslash-escaped Obsidian syntax — `[[link]]` → `\[\[link]]`, `![[img.png]]` → +> `!\[\[img.png]]`, `> [!tip]` → `> \[!tip]` — which Obsidian renders as literal text, +> so links, embedded images and callouts broke. It also rewrote ordinary prose +> (`figure [1]`, `~3 cm`, `snake_case`). **Fixed on `fix/obsidian-syntax-escaping`**: +> `src/editor/escapes.ts` removes every escape that re-parsing *proves* unnecessary. +> Branch 8 was framed as a feature; its *preservation* half was really a data-safety +> fix, the same way front matter (branch 10) was. Rendering and navigation remain branch 8. +> +> **Recommended order from here** (audience-weighted; supersedes the numbering for +> sequencing, not the numbering itself): +> 1. Ship `fix/obsidian-syntax-escaping` as a patch release — it is a §3 fix. +> 2. Merge **branch 6 (vault search)** — already built on `origin/feat/vault-search`. +> 3. **H1** vault-wide repair of notes damaged by earlier saves, and **H2** save +> compare-and-swap (closes the documented TOCTOU). +> 4. **H3 / H4** images in and out of notes the way Obsidian expects. +> 5. **H5** render what H0 now preserves: clickable wikilinks, image embeds, callouts. +> 6. **M1** templates → **M2** case log. +> 7. **M5** privacy kit, **M6** self-quiz, **M3** image viewing. +> 8. Branches 9 (tags) and 7 (palette); then **M7 / M8** study tools; then AI, local-first. +> +> Deprioritized for this audience: 13 (code highlighting), 16 (focus/typewriter), +> 17 (writing stats), and 28 (own mobile app — see §8: with faithful Obsidian syntax, +> Obsidian mobile on the same synced vault *is* the mobile story). + --- ## 2. How we work (recap — these don't change) @@ -286,6 +319,10 @@ editor something you keep using rather than something you tried. - [ ] **8. `feat/wikilinks-backlinks`** — `[[wikilink]]` with `[[`-autocomplete + a backlinks panel. The main thing that makes a folder of notes navigable, and it is the syntax Obsidian already uses, so vaults stay readable in both. + - *Preservation already done (H0, 2026-11-09):* `[[links]]` and `![[embeds]]` survive a + save byte-for-byte. Before that fix they were escaped on save, which made this a §3 + bug and not only a missing feature. What remains here is rendering, navigation, + autocomplete and backlinks (shared with H5). - *New crate:* `crates/linkgraph` — parse `[[links]]`, resolve to files, maintain a backlink index; updates on watcher events. - *Touches:* a Milkdown `$node`/`$inputRule` for the link + autocomplete (canonical, @@ -491,6 +528,197 @@ that has to land before a `v1.0` means anything. --- +### Track H — Hardening · *make "Obsidian-compatible" and "never loses a note" fully true* + +**Why:** each item below is a place where a claim in the README is weaker than it +reads. None is glamorous; all of them are what a heavy daily user hits first. + +- [x] **H0. `fix/obsidian-syntax-escaping`** — *(2026-11-09)* proof-checked escape + minimization in the canonical serializer, so wikilinks, embeds, callouts and plain + prose survive a save byte-for-byte. See CLAUDE.md §0. + - *Built:* `src/editor/escapes.ts`, wired in `serializer.ts`. + - *Gate:* `tests/escapes.test.ts` (search + 200-case property over the real parser) and + the `obsidianPreserved` / `escapesKept` classes in `roundtrip.test.ts`. + - *Open:* serialization now costs about one extra full parse (~180 ms at 12k words in + jsdom, paid on the debounced change listener). If very long notes feel sluggish, take + the whole-document proof off the dirty check, which only needs to know whether the text + differs from disk, and keep it on the write path. + - *Pre-existing quirks found while gating, not yet fixed:* a bare URL ending in `\` + gains a backslash on every save; remark moves a space out of a strikethrough + boundary. Both reproduce on `main` without H0. + +- [ ] **H1. `feat/repair-escaped-syntax`** — a vault-wide, **explicit** action that finds + notes damaged by pre-H0 saves (`\[\[`, `!\[\[`, `> \[!`) and repairs them. It shows the + list and per-file diff first, snapshots every file before writing, and writes atomically. + Never automatic (§3: no unrequested rewrites). Opening and saving a note in Toril already + repairs it, which is pinned as `repairsEscapedObsidianSyntax`. This is for the ones + nobody reopens. + - *Touches:* reuse `vaultsearch` for the scan; `snapshots` + `fsatomic` for the write. + - *Gate:* repair is idempotent; a note with no damage is byte-untouched. + +- [ ] **H2. `fix/save-compare-and-swap`** — close the documented TOCTOU (CLAUDE.md §5, + "Known limit of the guarantee"): `save_file` takes the expected base hash and refuses + the write unless the bytes on disk still match, inside the same Rust call. + - *Gate:* `fsatomic`-level test with a write landing between check and rename. + +- [ ] **H3. `feat/image-insertion`** — image input that works the way people try it: + - Drag an image file onto the editor and it is inserted. Today a drop only *opens* + files, and images are not on that allowlist, so they are silently dropped. + - Paste into an **unsaved** note: stage the image, and move it into place on first save. + - Paths with spaces, written as `<…>` or `%20`. Obsidian accepts both; CommonMark rejects + a raw space. + - Large-image guard: warn above a size threshold, and optionally downscale a *copy*. + - *Gate:* `imgasset` tests + a drop-allowlist case in `tests/paths.test.ts`. + +- [ ] **H4. `feat/obsidian-attachment-settings`** — read `.obsidian/app.json` + (`attachmentFolderPath`, `newLinkFormat`, `useMarkdownLinks`), so a pasted image lands + where Obsidian would put it and is written as `![[…]]` or `![](…)` to match the vault. + Read-only on `.obsidian/`; never write it. With no `.obsidian/`, keep `./assets/`. + +- [ ] **H5. `feat/obsidian-rendering`** — the rendering half of branch 8 plus its siblings: + - Clickable `[[wikilinks]]` (navigation only, resolved by `linkgraph`). + - Rendered `![[image.png]]` and `![[image.png|400]]`. + - Callouts `> [!type] Title`, including foldable `+`/`-`. + - The `==highlight==` mark. + + Each must stay a canonical node whose markdown form is *exactly* the Obsidian spelling, + so the H0 fixtures double as this branch's round-trip gate. + +- [ ] **H6. `feat/trash-and-purge`** — a trash browser (`list_trash` already has a command + and an `ipc.ts` wrapper) plus **permanent delete**. Permanent delete removes the + `.trash/` entry, the note's `history/` directory, and any `recovery.json` entry, after an + explicit confirm. Today nothing Toril deletes is ever really gone, and §M5 explains why + that matters. + +- [ ] **H7. Small items, each a few lines but each a stated promise:** + - `spellcheck="true"` on the editable, verified in WebView2. Also check whether the DOM + context menu suppresses the native spelling suggestions. + - Decide `app.security.csp` (CLAUDE.md §0, still open #3). + - Show `.html`/`.htm` in the sidebar, since HTML is advertised as first-class. + - Pull **branch 24 (per-tab undo)** forward. Losing undo on every tab switch is a daily + irritation, not a Movement V concern. + +- [ ] **H8. Reconsider PDF export (§7).** Students print and share study sheets, and + HTML → browser → "Save as PDF" is friction they notice. Re-weigh the per-platform + print APIs against the demand that now exists. Alternatively, a one-click "Print" + that opens the exported HTML in the default browser's print dialog would reduce the + friction without new unsafe FFI. + +--- + +### Track M — Medical & radiology students · *the audience that showed up* + +**Why:** a radiology note is a **case**: images first, a fixed structure (history → +findings → impression → differential → pearls), studied repeatedly, and sometimes +derived from material containing patient identifiers. Obsidian can approximate all of +this with five community plugins; Toril can do it out of the box, and the +local-first, no-account stance is a real fit for sensitive material. Keep every item +plain-markdown and Obsidian-readable: these users also open the vault in Obsidian, +often on a phone. + +- [ ] **M1. `feat/daily-notes-templates`** (= **branch 18**, pulled forward) with + **starter templates** shipped as ordinary `.md` files the user can edit: + - *Case note:* front matter for `modality`, `region`, `diagnosis`, `source`, + `rotation`, `date` and `tags`, then History / Technique / Findings / Impression / + Differential / Teaching points. + - *Structured report:* a Technique / Comparison / Findings / Impression skeleton in + the familiar structured-reporting shape. + - *Topic note:* Definition / Imaging features by modality / Mimics / Pearls / Pitfalls. + - *Differential table:* a GFM table template. + - *Gate:* `tests/templates.test.ts`. Every shipped template is also a round-trip + fixture, the same discipline as the welcome note. + +- [ ] **M2. `feat/case-log`** — a table view of front-matter properties across the vault, + sortable and filterable (*every CT chest case on my IR rotation*), with **CSV + export** for logbooks and applications. The properties strip already parses all three + formats. + - *Shape:* Rust returns raw front-matter blocks and the TS side parses them with the + *existing* `frontmatter-values.ts`. Do **not** add a Rust YAML crate: `serde_yaml` is + archived, so it fails §2. + - *Touches:* a `vault_properties` command (sibling of `vaultscan`) and + `src/ui/caselog.ts` in the rail or a tab. + - *Gate:* the splitter parity cases (TS ⇄ Rust) + a pure table-model test. + +- [ ] **M3. `feat/image-viewing`** — radiology is read on black: + - Click an image to open a lightbox with zoom and pan on a black background. + - Consecutive images in one paragraph lay out **side by side** (prior vs current, + T1/T2/FLAIR). This needs no new syntax, so it stays Obsidian-portable. + - Alt text shown as a caption. + - A low-luminance "reading room" theme. + - **Do not build a DICOM viewer**: screenshots are the medium, and DICOM parsing pulls in + a large, risky surface. + +- [ ] **M4. `feat/image-annotation`** — arrows, circles and short labels drawn on a + **copy** (`scan.annotated.png`); the original is never modified, and the note links + the copy. Canvas in the frontend, written through `imgasset`. Larger than it looks; + scope to three tools and undo. + +- [ ] **M5. `feat/privacy-kit`** — for material that may carry patient identifiers: + - **Strip image metadata** on insert (JPEG APPn/EXIF, PNG text chunks), done losslessly + in `imgasset` by dropping segments rather than re-encoding. + - **Crop on insert**, to cut burned-in names or record numbers off a screenshot before + it is ever written. + - **Permanent delete** (H6). Today a deleted note survives in `.trash/`, which sits + *inside* the synced vault, and in version history. + - A short `docs/` page on using Toril with clinical images: de-identify first, follow + your institution's policy, and think twice before a consumer cloud sync. **Never + claim HIPAA or GDPR compliance**: Toril is a local editor, and compliance is about the + whole workflow. + - *Gate:* `imgasset` tests that stripped output decodes identically and carries no + metadata segments. + +- [ ] **M6. `feat/self-quiz`** — unknown-case practice: + - Callouts folded with `-` (`> [!question]- What is the sign?`) render **collapsed** + and open on click. + - A per-note "hide answers" toggle collapses every foldable callout and `
`. + - Optionally hide sections by heading (Impression, Diagnosis) until revealed. + + It is purely view state, and the file never changes. Depends on H5's callout rendering. + +- [ ] **M7. `feat/flashcard-export`** — export `{{c1::cloze}}` and `Q::A` lines (the + Obsidian_to_Anki syntax, both already pinned as preserved) to an Anki-importable + **CSV**. An `.apkg` file needs SQLite, which means C (`rusqlite` bundles it) and fails + §2's C-free rule, so do not build it. Media cannot ride a CSV, so say so in the dialog. + Building spaced repetition *into* Toril is out of scope: students already use Anki, and + the job is to feed it. + +- [ ] **M8. `feat/review-queue`** — lightweight review without Anki: `review_next:` / + `last_reviewed:` front-matter dates, a "due today" list in the rail, and a "Reviewed" + action that updates the property. Uses typed rows, so it is only offered when the + block re-serializes byte-exact. + +- [ ] **M9. `feat/text-snippets`** — user-defined expansions (`;nlcxr` → a normal chest + radiograph paragraph), stored as a plain `.md` file in the vault so they sync and + stay editable. Insertion is a ProseMirror transaction, never raw text (§3.2). + +- [ ] **M10. Medical spelling** — depends on H7's spellcheck. A user dictionary that + persists, and optionally a bundled medical word list, **after a licence check** (several + common medical dictionaries are GPL or unclear; §2 applies to data too). + +- [ ] **M11. Pull forward branch 15 (Mermaid)** — decision trees and imaging algorithms + are a staple of these notes, far more than code highlighting or math. + +- [ ] **M12. AI, local-first for clinical content** — when branch 20 lands, default to + **Ollama**, and require an explicit per-vault opt-in, with a plain warning about patient + data, before any note goes to a cloud provider. Useful and low-risk: generate flashcards + or quiz questions from a note, summarize, outline. **Do not** build "suggest a diagnosis" + or differential features: the accuracy and liability profile is wrong for an editor. The + MCP bridge (branch 23) needs the same opt-in. + +- [ ] **M13. `feat/web-clip`** — paste from a reference site keeps structure, downloads + remote images into the attachment folder (offline study, no link rot), and records + `source:` in front matter for attribution. Sanitize as always (§3.3). + +- [ ] **M14. `feat/case-presentation`** — present a note as slides (split on `---` or H2), + image-first on black, arrow keys to advance: case conferences and teaching rounds. A + view, not a format, so there is nothing new to round-trip. + +- [ ] **M15. Portable build** — a zip that runs without installing, for locked-down + hospital and library PCs (WebView2 is already present on Windows 11). Settings and + history stay beside the executable when a marker file says so. + +--- + ### Parallel track — Toril-TUI follow-ups · *parked 2026-07-08* Not part of the desktop Movement ladder. **Toril-TUI** is a second, webview-free @@ -566,4 +794,41 @@ floor supports. Everything else is execution. - **Beta graduation bar:** which exact branches must be green to drop `-alpha`? (Proposed: all of Movements I–II.) - **Mobile scope for v1.0:** full edit parity, or read + light-edit first? (Lean - light-edit first; the sandbox makes full parity expensive.) + light-edit first; the sandbox makes full parity expensive.) **Re-opened 2026-11-09:** + with Obsidian syntax now preserved (H0), Obsidian mobile reading the same synced + vault may simply *be* the mobile answer — a quarter of work (branch 28) against a + free, mature client. Decide after H5 makes the desktop side render what mobile + writes. +- **How far to specialize for medical users?** Track M is mostly general-purpose + features with medical *defaults* (templates, case log, quiz, privacy). Keep it that + way: a templates system with a radiology starter pack, not a radiology app. Revisit + if the audience broadens or narrows. +- **Learn the audience without telemetry.** Toril does not phone home, and that stays. + Use an opt-in feedback channel (a GitHub Discussions category, or a short survey + linked from Help) and talk to a handful of the students directly before committing + to the larger M items (M2, M4). + +--- + +## 9. Value Toril does not offer today (unplaced — pull from here) + +Ideas that came out of the 2026-11-09 review, beyond Tracks H and M. Unscheduled and +in scope, held to the same rules: plain files, one serializer per format, all disk +I/O in Rust, healthy dependencies only. + +| Idea | Why it matters | Notes / constraints | +|---|---|---| +| **Split view** — two notes side by side | Comparing cases, writing from a source note | Today one shared editor swaps content (branch 24 is the prerequisite) | +| **PDF reader pane** with page-linked notes | Lecture slides and papers are where study notes come from | `pdf.js` (Mozilla, healthy); a link format like `[[deck.pdf#page=12]]` matches Obsidian | +| **Study-packet export** — many notes → one HTML (later PDF) | Printable revision packs, sharing with a study group | Reuses `mdhtml` + the sanitize chokepoint; ordering by folder or a list note | +| **Vault health check** | Broken `[[links]]`, orphaned attachments, oversized images, H1 damage | Read-only report; any fix is a separate, explicit, snapshotted action | +| **Citations** — Pandoc `[@key]` + a Zotero (Better BibTeX) JSON file | Research-track students and residents | `[@key]` already survives a save (H0); render a reference list on export | +| **Searchable image text (OCR)** | Find a case by text inside a screenshot | Use the OS engines (Windows.Media.Ocr, macOS Vision) through Rust bindings; Tesseract is C and fails §2. Store results outside the vault, like history | +| **Scheduled backup** — vault + history to a second folder or zip | Version history lives in the app config dir; a disk loss takes both | Atomic, additive, opt-in; never into the vault itself | +| **Import cleanup** — Notion, OneNote, Evernote exports | Students arrive from OneNote and Notion | Notion's escape-heavy markdown is now normalized by H0 on first edit; an explicit batch import is the remaining piece | +| **Link previews** for reference URLs | Context on hover | Any network fetch is a privacy decision: opt-in, no telemetry, nothing sent but the URL the user wrote | +| **Shared study vault guidance** | Groups already share a synced folder | The conflict machinery (branch 4) is built for this; it needs documenting more than code | + +**Considered and deliberately left out:** a graph view (rarely used after the first +week; backlinks carry the value), a plugin system (README: "no plans"), and an in-app +spaced-repetition engine (feed Anki instead — M7). diff --git a/src/editor/escapes.ts b/src/editor/escapes.ts new file mode 100644 index 0000000..4c2afe5 --- /dev/null +++ b/src/editor/escapes.ts @@ -0,0 +1,235 @@ +// Verified escape minimization for the canonical serializer (CLAUDE.md §3.2). +// +// remark-stringify escapes defensively: every `[`, `*`, `_`, `~` in prose gets a +// backslash whether or not it could ever be read as syntax. That is safe but it +// rewrites the user's file — and for Obsidian vaults it is *destructive*: +// `[[Note]]` becomes `\[\[Note]]`, which Obsidian renders as literal text, so +// every wikilink, image embed (`![[x.png]]`) and callout (`> [!tip]`) in an +// edited note silently breaks. +// +// This module removes the escapes that are not needed, and it decides "not +// needed" by **proof, not by rule**: a removal is kept only if the candidate +// markdown parses to exactly the same document as the fully escaped output. The +// escaping rules of CommonMark + GFM (flanking delimiter runs, reference labels, +// task-list markers, list bullets at line start, autolink-literal boundaries…) +// are never re-implemented here, so they cannot be re-implemented wrongly. The +// worst a bug in candidate extraction can do is cost a parse. +// +// Pure — no DOM, no Milkdown — so the search is gated headlessly with a fake +// `sameDoc` (tests/escapes.test.ts); `serializer.ts` supplies the real parser. + +/** One removable piece of the serializer's output: the indices to delete. */ +interface Candidate { + readonly drop: readonly number[]; +} + +/** + * Upper bound on whole-document parses in the fallback search. It only bounds + * pathological input — anything not yet verified when it is hit keeps its + * escape, which is always safe. + */ +const MAX_PARSES = 64; + +const ASCII_PUNCT = /[!-/:-@[-`{-~]/; + +// ``, ``, `` and `` — the forms remark emits +// for a link whose text equals its URL. Removing both brackets yields the bare +// form Obsidian writes; GFM's autolink-literal parse decides if that holds. +const AUTOLINK = /<((?:https?:\/\/|mailto:|www\.)[^\s<>]+|[^\s@<>]+@[^\s@<>]+\.[^\s<>]+)>/g; + +// A fence opener/closer, allowing for blockquote and list-item prefixes. +const FENCE = /^[ \t>]*(?:(?:[-+*]|\d{1,9}[.)])[ \t]+)?(`{3,}|~{3,})/; + +/** + * Collect candidate removals outside code. Code is skipped only to save parses: + * a backslash inside a code span or fence is content, so removing it would be + * rejected by verification anyway. + */ +export function findCandidates(md: string): Candidate[] { + const out: Candidate[] = []; + let fence: string | null = null; + let lineStart = 0; + + for (const line of md.split("\n")) { + const m = FENCE.exec(line); + if (fence !== null) { + if (m && m[1][0] === fence[0] && m[1].length >= fence.length) fence = null; + } else if (m) { + fence = m[1]; + } else { + scanLine(line, lineStart, out); + } + lineStart += line.length + 1; + } + return out; +} + +function scanLine(line: string, offset: number, out: Candidate[]): void { + const code = codeSpans(line); + const inCode = (i: number): boolean => code.some(([s, e]) => i >= s && i < e); + + for (let i = 0; i < line.length; i++) { + if (line[i] !== "\\") continue; + const next = line[i + 1]; + if (next === undefined || !ASCII_PUNCT.test(next)) continue; + if (!inCode(i) && next !== "\\") out.push({ drop: [offset + i] }); + i++; // the escaped character is consumed either way (`\\` pairs too) + } + + AUTOLINK.lastIndex = 0; + for (let m = AUTOLINK.exec(line); m !== null; m = AUTOLINK.exec(line)) { + const start = m.index; + const end = start + m[0].length - 1; + if (inCode(start) || line[start - 1] === "\\") continue; + out.push({ drop: [offset + start, offset + end] }); + } +} + +/** [start, end) of each inline code span on a line (backtick runs of equal length). */ +function codeSpans(line: string): Array<[number, number]> { + const spans: Array<[number, number]> = []; + let i = 0; + while (i < line.length) { + if (line[i] !== "`" || (i > 0 && line[i - 1] === "\\")) { + i++; + continue; + } + let n = 0; + while (line[i + n] === "`") n++; + const run = "`".repeat(n); + let j = line.indexOf(run, i + n); + while (j !== -1 && (line[j + n] === "`" || line[j - 1] === "`")) { + j = line.indexOf(run, j + 1); + } + if (j === -1) { + i += n; + continue; + } + spans.push([i, j + n]); + i = j + n; + } + return spans; +} + +function apply(md: string, candidates: readonly Candidate[], chosen: Iterable): string { + const drop = new Set(); + for (const c of chosen) for (const i of candidates[c].drop) drop.add(i); + if (drop.size === 0) return md; + let s = ""; + for (let i = 0; i < md.length; i++) if (!drop.has(i)) s += md[i]; + return s; +} + +/** + * Remove every candidate whose removal `same` accepts, verifying each accepted + * group *jointly* with all previously accepted ones — so interacting removals + * (`\*a\*`: either star alone is literal, both together are emphasis) can never + * both slip through. Groups are bisected on failure; once `budget` parses are + * spent, everything still unverified keeps its escape, which is always safe. + */ +function search(md: string, same: (candidate: string) => boolean, budget: number): string { + const candidates = findCandidates(md); + if (candidates.length === 0) return md; + + const accepted: number[] = []; + let parses = 0; + const test = (group: readonly number[]): boolean => { + if (parses >= budget) return false; + parses++; + try { + return same(apply(md, candidates, [...accepted, ...group])); + } catch { + return false; // a parser failure is never evidence of equality + } + }; + const bisect = (group: readonly number[]): void => { + if (group.length === 0) return; + if (test(group)) { + accepted.push(...group); + return; + } + if (group.length === 1) return; + const mid = group.length >> 1; + bisect(group.slice(0, mid)); + bisect(group.slice(mid)); + }; + bisect(candidates.map((_, i) => i)); + return apply(md, candidates, accepted); +} + +/** + * Split markdown into top-level blocks at blank lines outside fences, keeping + * every separator so `join("")` reproduces the input exactly. A block is a + * *unit of cheap verification*, not a claim that blocks parse independently — + * they do not always (loose lists, reference definitions), which is why the + * whole document is verified again afterwards. + */ +export function splitBlocks(md: string): string[] { + const parts: string[] = []; + let fence: string | null = null; + let current = ""; + let blank = false; + for (const line of md.split(/(?<=\n)/)) { + const isBlank = fence === null && /^[ \t]*\n?$/.test(line); + if (isBlank !== blank && current !== "") { + parts.push(current); + current = ""; + } + blank = isBlank; + current += line; + const m = FENCE.exec(line); + if (fence !== null) { + if (m && m[1][0] === fence[0] && m[1].length >= fence.length) fence = null; + } else if (m) { + fence = m[1]; + } + } + if (current !== "") parts.push(current); + return parts; +} + +/** Budgets: per block (small parses), and for the whole-document fallback. */ +const BLOCK_PARSES = 24; + +// Block → minimized block. Minimizing a block depends only on its own text (it +// is verified standalone, against itself), so between keystrokes every block but +// the one being edited is a cache hit and costs no parse at all. +const blockCache = new Map(); +const BLOCK_CACHE_MAX = 2000; + +/** + * Return `md` with every escape (and autolink bracket pair) removed that can be + * removed without changing the parsed document. + * + * `same(candidate, reference)` must answer whether `candidate` parses to the + * same document as `reference`. Two passes: + * + * 1. **Per block**, each block verified standalone against its own escaped text + * — small parses, memoized. + * 2. **Whole document**, one parse: the joined result against `md`. This is the + * proof that matters; pass 1 only proposes. If it fails (a block whose + * meaning depends on another — a `[label]` that a definition elsewhere turns + * into a link, a loose list split across blocks), fall back to searching the + * whole document directly, within `MAX_PARSES`. + */ +export function minimizeEscapes(md: string, same: (candidate: string, reference: string) => boolean): string { + if (findCandidates(md).length === 0) return md; + + const blocks = splitBlocks(md).map((block) => { + const hit = blockCache.get(block); + if (hit !== undefined) return hit; + const minimized = search(block, (c) => same(c, block), BLOCK_PARSES); + if (blockCache.size >= BLOCK_CACHE_MAX) blockCache.clear(); + blockCache.set(block, minimized); + return minimized; + }); + const joined = blocks.join(""); + if (joined === md) return md; + + try { + if (same(joined, md)) return joined; + } catch { + // fall through to the whole-document search + } + return search(md, (c) => same(c, md), MAX_PARSES); +} diff --git a/src/editor/serializer.ts b/src/editor/serializer.ts index da9eb6a..ef3d01b 100644 --- a/src/editor/serializer.ts +++ b/src/editor/serializer.ts @@ -7,11 +7,49 @@ // ProseMirror), so we inherit its tested round-trip behavior rather than // hand-rolling a schema (§11). import type { Editor } from "@milkdown/kit/core"; -import { getMarkdown, replaceAll } from "@milkdown/kit/utils"; +import { editorViewCtx, parserCtx, serializerCtx } from "@milkdown/kit/core"; +import type { Node } from "@milkdown/kit/prose/model"; +import { replaceAll } from "@milkdown/kit/utils"; +import { minimizeEscapes } from "./escapes"; -/** Serialize the editor's current document to canonical markdown. */ +/** + * Serialize the editor's current document to canonical markdown. + * + * remark escapes defensively, which turns Obsidian's `[[wikilinks]]`, + * `![[embeds]]` and `> [!callouts]` into literal text. `minimizeEscapes` takes + * back every escape whose removal provably parses to the same document. + * + * For the whole document, a candidate is accepted if it loads as the **live** + * document (the ideal, and one parse) or, failing that, as the fully escaped + * output would — the exact pre-minimization behavior, which differs from the + * live doc only where remark's round trip is already lossy. + */ export function docToMarkdown(editor: Editor): string { - return editor.action(getMarkdown()); + return editor.action((ctx) => { + const serializer = ctx.get(serializerCtx); + const parser = ctx.get(parserCtx); + const live = ctx.get(editorViewCtx).state.doc; + const escaped = serializer(live); + + let escapedDoc: Node | null = null; + let lastRef: string | null = null; + let lastRefDoc: Node | null = null; + return minimizeEscapes(escaped, (candidate, reference) => { + const parsed = parser(candidate); + if (reference === escaped) { + if (parsed.eq(live)) return true; + escapedDoc ??= parser(escaped); + return parsed.eq(escapedDoc); + } + // A block, verified standalone. Bisection re-tests against the same + // reference many times, so keep its parse. + if (reference !== lastRef) { + lastRef = reference; + lastRefDoc = parser(reference); + } + return lastRefDoc !== null && parsed.eq(lastRefDoc); + }); + }); } /** Replace the editor's entire document with the result of parsing `markdown`. */ diff --git a/src/welcome.ts b/src/welcome.ts index 4cd45d5..6d36df9 100644 --- a/src/welcome.ts +++ b/src/welcome.ts @@ -60,7 +60,7 @@ also records a version you can go back to, and if a file changes underneath you --- -Ctrl-click a link to open it in your browser: +Ctrl-click a link to open it in your browser: https://github.com/kovirlabs/toril `; export const EMPTY = `# Untitled diff --git a/tests/escapes.test.ts b/tests/escapes.test.ts new file mode 100644 index 0000000..a042c4e --- /dev/null +++ b/tests/escapes.test.ts @@ -0,0 +1,189 @@ +// Gate for src/editor/escapes.ts — verified escape minimization (CLAUDE.md §3.2). +// +// Two halves. The pure half drives `minimizeEscapes` with a fake `sameDoc`, so +// the search itself is pinned: joint verification, bisection, the parse cap, +// and that a throwing parser is never read as "equal". The property half runs +// the real canonical editor over randomized prose built from exactly the +// characters remark escapes, and asserts the only thing that matters: the saved +// file loads as the same document, and saving it again changes nothing. +import { describe, expect, it } from "vitest"; +import { Editor, defaultValueCtx, editorViewCtx, rootCtx, serializerCtx } from "@milkdown/kit/core"; +import { commonmark } from "@milkdown/kit/preset/commonmark"; +import { gfm } from "@milkdown/kit/preset/gfm"; +import { emoji } from "@milkdown/plugin-emoji"; +import { useCanonical } from "../src/editor/canonical"; +import { findCandidates, minimizeEscapes, splitBlocks } from "../src/editor/escapes"; +import { docToMarkdown } from "../src/editor/serializer"; + +// A tiny deterministic PRNG so a failure is reproducible from its seed. +function rng(seed: number): () => number { + let s = seed >>> 0; + return () => { + s = (s * 1664525 + 1013904223) >>> 0; + return s / 2 ** 32; + }; +} + +describe("findCandidates", () => { + it("finds escapes and autolink bracket pairs in prose", () => { + expect(findCandidates("a \\[b] \\* \n")).toEqual([ + { drop: [2] }, + { drop: [7] }, + { drop: [10, 22] }, + ]); + }); + + it("never offers an escaped backslash", () => { + expect(findCandidates("x\\\\y and \\\\[z]")).toEqual([]); + }); + + it("skips inline code and fenced code", () => { + const md = "`a\\[b` ``c\\*`d``\n```\n\\[lit]\n```\n> ~~~\n> \\*q\n> ~~~\nafter \\_\n"; + const found = findCandidates(md); + expect(found).toHaveLength(1); + expect(md[found[0].drop[0] + 1]).toBe("_"); + }); + + it("ignores a backslash before a non-punctuation character", () => { + expect(findCandidates("C:\\Users\\me\n")).toEqual([]); + }); +}); + +describe("splitBlocks", () => { + it("reproduces its input exactly", () => { + const rand = rng(7); + const bits = ["a", "\n", "\n\n", " \n", "```\n", "~~~\n", "> x\n", "- y\n", " "]; + for (let n = 0; n < 500; n++) { + const md = Array.from({ length: Math.floor(rand() * 20) }, () => bits[Math.floor(rand() * bits.length)]).join(""); + expect(splitBlocks(md).join("")).toBe(md); + } + }); + + it("keeps a fence with blank lines inside as one block", () => { + expect(splitBlocks("p\n\n```\na\n\nb\n```\n\nq\n")).toEqual(["p\n", "\n", "```\na\n\nb\n```\n", "\n", "q\n"]); + }); +}); + +describe("minimizeEscapes (search)", () => { + // Every case uses text no other case uses: the block cache is keyed on text, + // so reusing a block would read another case's verdict. + it("returns the input untouched, without parsing, when there is nothing to try", () => { + let calls = 0; + expect(minimizeEscapes("plain text\n", () => (calls++, true))).toBe("plain text\n"); + expect(calls).toBe(0); + }); + + it("costs one block parse plus one document parse when every removal is valid", () => { + let calls = 0; + const out = minimizeEscapes("\\[\\[a]] \\_b\\_ \\~c\n", () => (calls++, true)); + expect(out).toBe("[[a]] _b_ ~c\n"); + expect(calls).toBe(2); + }); + + it("re-verifies only the document when every block is cached", () => { + const md = "\\[\\[cached]]\n\n\\_also\\_\n"; + minimizeEscapes(md, () => true); + let calls = 0; + expect(minimizeEscapes(md, () => (calls++, true))).toBe("[[cached]]\n\n_also_\n"); + expect(calls).toBe(1); + }); + + it("verifies removals jointly — interacting escapes never both go", () => { + // Either star alone is literal; both together would be emphasis. + const out = minimizeEscapes("\\*joint\\*\n", (c) => c.includes("\\*")); + expect(out.match(/\\\*/g)).toHaveLength(1); + }); + + it("bisects to keep only the escapes that are needed", () => { + const out = minimizeEscapes("\\[1] \\[2] \\- \\[3]\n", (c) => c.includes("\\-")); + expect(out).toBe("[1] [2] \\- [3]\n"); + }); + + it("falls back to a whole-document search when blocks pass but the document does not", () => { + // Models a definition in another block turning `[ref]` into a link: standalone + // the bracket is literal, in the document it is not. + const md = "\\[ref] and \\_x\\_\n\n[ref]: https://e.x\n"; + const same = (c: string, r: string): boolean => r !== md || c.includes("\\[ref]"); + expect(minimizeEscapes(md, same)).toBe("\\[ref] and _x_\n\n[ref]: https://e.x\n"); + }); + + it("treats a throwing parser as inequality", () => { + const boom = (): boolean => { + throw new Error("boom"); + }; + expect(minimizeEscapes("\\[thrown]\n", boom)).toBe("\\[thrown]\n"); + }); + + it("bounds the number of parses on pathological input", () => { + let calls = 0; + const md = Array.from({ length: 500 }, (_, i) => `\\[p${i}]`).join(" ") + "\n"; + const out = minimizeEscapes(md, () => (calls++, false)); + expect(calls).toBeLessThanOrEqual(24 + 64); + expect(out).toBe(md); // unverified ⇒ escape kept, which is always safe + }); +}); + +async function makeEditor(md: string): Promise { + const root = document.createElement("div"); + document.body.appendChild(root); + return useCanonical( + Editor.make() + .config((ctx) => { + ctx.set(rootCtx, root); + ctx.set(defaultValueCtx, md); + }) + .use(commonmark) + .use(gfm) + .use(emoji), + ).create(); +} + +const docJson = (e: Editor): string => + JSON.stringify(e.action((ctx) => ctx.get(editorViewCtx).state.doc.toJSON())); + +const TOKENS = [ + "a", "b", "word", " ", " ", " ", "*", "_", "~", "[", "]", "(", ")", "!", "#", + "-", "+", "1.", "<", ">", "`", "|", ":", "[[x]]", "![[i.png]]", "[!tip]", + "https://e.x/a_b", "\\", "&", "^", "=", "%%", +]; + +/** What the serializer emits before minimization — the pre-fix behavior. */ +const fullyEscaped = (e: Editor): string => + e.action((ctx) => ctx.get(serializerCtx)(ctx.get(editorViewCtx).state.doc)); + +describe("minimizeEscapes (property, real parser)", () => { + // The guarantee is relative, and deliberately so: the minimized file loads as + // EXACTLY the document the fully escaped file would. Random input also hits a + // few upstream quirks that are lossy or unstable with or without this module + // (remark trims a space out of a strikethrough boundary; a bare URL ending in + // `\` grows a backslash per save — CLAUDE.md §0). Comparing against the + // escaped baseline gates *this* change without being masked by those. + it("never loads differently from the fully escaped output, and is no less stable", async () => { + const rand = rng(20261109); + let minimizedSomething = 0; + for (let n = 0; n < 200; n++) { + const lines = Array.from({ length: 1 + Math.floor(rand() * 3) }, () => + Array.from({ length: 1 + Math.floor(rand() * 10) }, () => TOKENS[Math.floor(rand() * TOKENS.length)]).join(""), + ); + const input = lines.join("\n\n") + "\n"; + const label = `case ${n}: ${JSON.stringify(input)}`; + + const first = await makeEditor(input); + const escaped = fullyEscaped(first); + const saved = docToMarkdown(first); + if (saved !== escaped) minimizedSomething++; + + const fromSaved = await makeEditor(saved); + const fromEscaped = await makeEditor(escaped); + expect(docJson(fromSaved), label).toBe(docJson(fromEscaped)); + + // Stability: wherever the old serializer was idempotent, so is the new one. + if (fullyEscaped(fromEscaped) === escaped) { + expect(docToMarkdown(fromSaved), label).toBe(saved); + } + await Promise.all([first.destroy(), fromSaved.destroy(), fromEscaped.destroy()]); + } + // Guard against a vacuous pass (e.g. a generator that stops producing escapes). + expect(minimizedSomething).toBeGreaterThan(100); + }, 120_000); +}); diff --git a/tests/roundtrip.test.ts b/tests/roundtrip.test.ts index 7120a29..710870c 100644 --- a/tests/roundtrip.test.ts +++ b/tests/roundtrip.test.ts @@ -89,6 +89,53 @@ const preserved: Record = { thematicBreak: "Above.\n\n---\n\nBelow.\n", // deliberate overlap with `fixtures` asteriskEmphasis: "Some *italic* and **bold** text.\n", underscoreEmphasis: "Some _italic_ and __bold__ text.\n", + // Escapes are emitted only where the parse proves them necessary + // (src/editor/escapes.ts). Before that, every one of these gained backslashes. + literalAsterisk: "A literal asterisk * and an underscore_in_word here.\n", + bracketCitation: "See figure [1] and [a, b].\n", + approximateTilde: "50% stenosis, ~3 cm, 5 * 3 mm.\n", + bareAutolink: "Visit https://example.com today.\n", +}; + +// Obsidian-specific syntax. Toril does not *render* most of it yet, but it must +// never break it: before src/editor/escapes.ts, saving an edited note rewrote +// `[[Note]]` as `\[\[Note]]`, which Obsidian shows as literal text — every +// wikilink, image embed and callout in that note silently stopped working. +const obsidianPreserved: Record = { + wikilink: "See [[Pneumothorax]] and [[Pneumothorax#Signs|signs]].\n", + wikilinkBlockRef: "As in [[Case 12^finding]].\n", + adjacentWikilinks: "[[a]][[b]]\n", + wikilinkInHeading: "# Heading with [[Link]]\n", + wikilinkInBold: "**bold [[Link]]** text\n", + wikilinkInList: "- item [[Link]]\n- ![[img.png]]\n", + imageEmbed: "![[Pasted image 20261101.png]]\n", + imageEmbedSized: "![[cxr.png|400]]\n", + noteEmbed: "![[Chest#Signs]]\n", + callout: "> [!tip] Pearl\n> Deep sulcus sign on supine film.\n", + calloutFoldOpen: "> [!note]+ Title\n> body\n", + calloutFoldClosed: "> [!question]- What is the sign?\n> Answer.\n", + nestedCallout: "> > [!tip] nested\n> > x\n", + highlight: "The ==silhouette sign== matters.\n", + comment: "Text %%hidden%% here.\n", + blockId: "A key finding. ^finding1\n", + nestedTag: "Case of #chest/pneumothorax here.\n", + sizedImage: "![cxr|300](assets/cxr.png)\n", + ankiCloze: "The {{c1::deep sulcus}} sign.\n", + ankiInline: "Q:: What is X?\nA:: Y\n", + inlineMath: "Inline $E=mc^2$ math.\n", + blockMath: "$$\nx^2\n$$\n", +}; + +// The other half of the escape rule: where an escape IS needed, it stays. Each +// of these would change meaning without its backslash (task item, heading, +// ordered list, bullet, blockquote, raw HTML), so the output must keep it. +const escapesKept: Record = { + taskMarkerInList: "- \\[ ] literal\n", + headingMarker: "\\# not heading\n", + orderedMarker: "1\\. not list\n", + bulletMarker: "\\- not bullet\n", + quoteMarker: "\\> not quote\n", + rawHtml: "\\
text\n", }; // Input that Toril legitimately rewrites. Each entry pins the EXACT output, so @@ -109,9 +156,17 @@ const normalized: Record = { asteriskRule: ["Above.\n\n***\n\nBelow.\n", "Above.\n\n---\n\nBelow.\n"], underscoreRule: ["Above.\n\n___\n\nBelow.\n", "Above.\n\n---\n\nBelow.\n"], tablePadding: ["| A | B |\n|---|---|\n| 1 | 2 |\n", "| A | B |\n| - | - |\n| 1 | 2 |\n"], - intrawordUnderscore: [ - "A literal asterisk \\* and an underscore_in_word here.\n", + // Superfluous escapes are dropped — the reverse of the old normalization, + // which *added* them. Rendering is identical; only unneeded backslashes go. + superfluousEscapes: [ "A literal asterisk \\* and an underscore\\_in\\_word here.\n", + "A literal asterisk * and an underscore_in_word here.\n", + ], + // Obsidian table cells keep their escaped pipe; the column padding is computed + // before minimization, so it may run a character wide. + wikilinkAliasInTable: [ + "| a | b |\n| - | - |\n| [[x\\|y]] | z |\n", + "| a | b |\n| ---------- | - |\n| [[x\\|y]] | z |\n", ], emojiShortcode: ["Hello :smile: world\n", "Hello 😄 world\n"], // An UNUSED link reference definition is DELETED — the one place the rewrite @@ -123,7 +178,13 @@ const normalized: Record = { // this single normalization dominates the conflict rate that feat/sync-coexistence // reasons about, far more than any construct-level reformat below. crlfLineEndings: ["- one\r\n- two\r\n", "- one\n- two\n"], - bareAutolink: ["Visit https://example.com today.\n", "Visit today.\n"], + // A note damaged by an earlier Toril save (CHANGELOG, Unreleased) is repaired by + // the next one — the claim the release notes make to affected users. + repairsEscapedObsidianSyntax: [ + "See \\[\\[Note]] and !\\[\\[scan.png]].\n\n> \\[!tip] Pearl\n", + "See [[Note]] and ![[scan.png]].\n\n> [!tip] Pearl\n", + ], + angleAutolink: ["Visit today.\n", "Visit https://example.com today.\n"], plusBullets: ["+ one\n+ two\n", "- one\n- two\n"], parenOrderedMarker: ["1) one\n2) two\n", "1. one\n2. two\n"], overIndentedNesting: ["- Parent\n - Child\n", "- Parent\n - Child\n"], @@ -192,6 +253,18 @@ describe("round-trip fidelity (Phase 1 gate)", () => { }); } + for (const [name, md] of Object.entries(obsidianPreserved)) { + it(`preserves Obsidian syntax: ${name}`, async () => { + expect(await roundtrip(md)).toBe(md); + }); + } + + for (const [name, md] of Object.entries(escapesKept)) { + it(`keeps a necessary escape: ${name}`, async () => { + expect(await roundtrip(md)).toBe(md); + }); + } + for (const [name, md] of Object.entries(preserved)) { it(`preserves human-authored input: ${name}`, async () => { const once = await roundtrip(md); From 5c85aba2f572e21ad15d84fc6f8f5b9bc2f1dab3 Mon Sep 17 00:00:00 2001 From: Evan Gress <106449014+evangress@users.noreply.github.com> Date: Mon, 9 Nov 2026 21:48:24 -0500 Subject: [PATCH 2/4] docs(roadmap): record findings from a real-vault round-trip of the escape fix Co-Authored-By: Claude Opus 5.5 --- ROADMAP.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/ROADMAP.md b/ROADMAP.md index 8790b33..1c1d696 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -547,6 +547,35 @@ reads. None is glamorous; all of them are what a heavy daily user hits first. gains a backslash on every save; remark moves a space out of a strikethrough boundary. Both reproduce on `main` without H0. +- [ ] **H0b. Findings from a real vault (2026-11-09).** H0 was checked against a + 179-note personal Obsidian vault: each note was split, round-tripped and rejoined + exactly as `main.ts` does, on a copy. Results: + - Before H0, saving broke links in 23 notes: 352 wikilinks/embeds/callout markers + came back as 296. + - After H0, no note lost a link, no note loads differently, and no note became + unstable. + - The only differences between `main` and H0 across the whole vault are 1,175 removed + backslashes and 81 removed autolink `<…>` pairs. + + The same run surfaced rewrites and a crash that predate H0. Each one is a §3 item, in + order of reach: + - **A note containing `®`, `©` or `™` fails to load** in the jsdom gate + (`RangeError: Invalid array passed to renderSpec`). It comes from `@milkdown/plugin-emoji` + treating those characters as emoji: the same input loads fine without the plugin. + **Confirm in WebView2/WKWebView first.** If it reproduces there, it is the most urgent + item on this list: medical notes are full of `®` drug and contrast-agent names. + - **An empty list item `- ` is saved as `-
`** in 56 of 178 notes. Obsidian users + routinely leave a trailing empty bullet, and Toril writes raw HTML into the file. + - Blank-line normalization, in about 130 notes. + - Trailing whitespace stripped, in 71 notes. This is not in CLAUDE.md's + normalization list. + - Table re-padding, in 31 notes (already pinned as `tablePadding`). + - Tab-indented nested lists rewritten to two spaces. + - Emphasis split around inline code: `_Source: \`path\`_` → `_Source:_ _\`path\`_`. + + Each needs a fixture pinning today's behavior before it is changed, as with every + `normalized` entry. + - [ ] **H1. `feat/repair-escaped-syntax`** — a vault-wide, **explicit** action that finds notes damaged by pre-H0 saves (`\[\[`, `!\[\[`, `> \[!`) and repairs them. It shows the list and per-file diff first, snapshots every file before writing, and writes atomically. From 62f74c77ab7c000abe7e4e2e3f78b6b38319aaec Mon Sep 17 00:00:00 2001 From: Evan Gress <106449014+evangress@users.noreply.github.com> Date: Mon, 9 Nov 2026 22:27:30 -0500 Subject: [PATCH 3/4] =?UTF-8?q?fix(editor):=20keep=20=C2=A9/=C2=AE/?= =?UTF-8?q?=E2=84=A2=20as=20text=20and=20contain=20loads=20that=20throw?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Any note containing ©, ® or ™ failed to open with "RangeError: Invalid array passed to renderSpec". @milkdown/plugin-emoji's remark step splits text on emoji-regex, which matches those symbols, but twemoji leaves them as plain text, so the resulting emoji node's toDOM hands ProseMirror a Text node (rejected since prosemirror-model 1.25.5). Its toMarkdown also writes only an 's title, so a save would have deleted the symbols: "A © B" became "A B". plugin-emoji 7.23.0 fixes the crash but not the deletion, so upgrading alone would trade an open error for silent loss. src/editor/emoji.ts is now the one emoji stack (app and gates import it) and adds a remark step that turns any emoji node twemoji did not draw as an image back into text. Emoji-presentation forms (©️ ®️ ™️) are unchanged. The failed open also left the new tab active while the editor still showed the previous note, with its state already replaced by the half-loaded document; Save needs no dirty flag, so it could write that over the file. TabManager.setActive now restores and reloads the previous tab when onActivate throws, never flushes into the failed tab, and rethrows. openPath drops the never-activated tab, loadIntoEditor resets `loading` in a finally, and sidebar/second-launch/drop opens report the failure instead of rejecting unhandled. Gates: tests/emoji.test.ts, two round-trip `preserved` fixtures that mount a view, and four tabs tests for a throwing activation; all red before. Verified on-device against ~/toril-test-vault. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 13 ++++++ CLAUDE.md | 10 ++++- src/editor/emoji.ts | 68 ++++++++++++++++++++++++++++++ src/editor/milkdown.ts | 2 +- src/main.ts | 62 ++++++++++++++++++++++++--- src/ui/tabs.ts | 21 +++++++++- tests/emoji.test.ts | 81 ++++++++++++++++++++++++++++++++++++ tests/escapes.test.ts | 2 +- tests/html-roundtrip.test.ts | 2 +- tests/roundtrip.test.ts | 8 +++- tests/tabs.test.ts | 68 ++++++++++++++++++++++++++++++ tests/toolbar.test.ts | 2 +- 12 files changed, 326 insertions(+), 13 deletions(-) create mode 100644 src/editor/emoji.ts create mode 100644 tests/emoji.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 83f1103..04598f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,19 @@ GitHub Release notes plus the commits that shipped in it. turn into a heading or list. Every save checks this by re-reading what it is about to write and comparing it with your note, so no backslash is removed where it is needed. +- **Notes containing ©, ® or ™ open again, and those symbols are kept when you save.** + Any note with one of them failed to open with *"RangeError: Invalid array passed to + renderSpec"*. The emoji support treated the three symbols as emoji it could not draw, and + a save would also have deleted them: `Omnipaque® and Visipaque™, © 2026` would have been + written as `Omnipaque and Visipaque, 2026`. They are now ordinary text. The emoji + versions (©️ ®️ ™️, as an emoji keyboard or `:tm:` types them) are unchanged. + +- **A note that fails to open no longer leaves a broken tab behind.** After a failed open, + the new tab stayed selected while the editor still showed the previous note, and + pressing Save could write the wrong text into the file that had failed to open. Toril + now stays on the note you were in and reports the error, including when you open a note + from the sidebar, which previously failed without any message. + ## [v1.1.1] — 2026-10-15 ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index 09d8f07..5d01912 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -364,6 +364,7 @@ toril/ │ │ ├── milkdown.ts # WYSIWYG setup + plugins (incl. clipboard-image paste, §6) │ │ ├── serializer.ts # the ONE markdown <-> doc converter (§3.2) │ │ ├── escapes.ts # proof-checked escape minimization, used only by serializer.ts +│ │ ├── emoji.ts # the ONE emoji stack: upstream + ©/®/™ kept as text (§6) │ │ ├── html-serializer.ts # the ONE HTML <-> doc converter (§3.2/§3.3) │ │ └── html-constructs.ts # richer HTML-only schema: callout/details/dl/mark/sub/sup (§6) │ ├── ui/ @@ -635,7 +636,7 @@ frontend never touches the filesystem directly; it asks via `invoke()`. | WYSIWYG core | `@milkdown/core` + `@milkdown/preset-commonmark` | | GFM (tables, task lists, strikethrough) | `@milkdown/preset-gfm` | | Math (KaTeX) | **Deferred** — `@milkdown/plugin-math` is deprecated; omitted until a maintained option exists (§0/§8) | -| Emoji shortcodes | `@milkdown/plugin-emoji` | +| Emoji shortcodes | `@milkdown/plugin-emoji`, assembled **only** in `src/editor/emoji.ts` (app and every gate import `emoji` from there). That module adds one remark step: an emoji node twemoji did not draw as an `` (bare ©, ®, ™ — `emoji-regex` matches them, twemoji leaves them as text) goes back to plain text. Upstream made them nodes that crash the view on load (prosemirror-model ≥ 1.25.5 rejects the Text node `toDOM` returns) **and** serialize to nothing. **Do not upgrade to 7.23.0 thinking it fixes this**: it fixes the crash, not the deletion on save | | Inline / slash shortcuts | `@milkdown/plugin-slash` + keymap config | | Formatting toolbar | `toolbar.ts` → Milkdown commands via `callCommand` (the few command-less items use a plain ProseMirror transaction); **never inserts raw markdown text** (§3.2). Buttons reflect active state via `activeState()`. The pure command layer is exported separately from the DOM so the gate tests it headlessly. *Underline omitted by design* (no markdown form). *Front-matter button deferred* (not lossless yet). | | Front matter | **Not a Milkdown plugin, by design** — `src/editor/frontmatter.ts` splits the block off at the load/serialize boundary so it never enters the ProseMirror doc (§3). Edited in `src/ui/properties.ts`, a collapsible strip above the writing surface: typed rows for a block that provably re-serializes byte-exact, raw text for anything else | @@ -760,6 +761,13 @@ Phases 0–3 are complete and Phase 4 (polish) is in progress; the shipped detai in the **future** fails *open* — a corrected clock must not disable update checks forever with no symptom. What it cannot cover is anything downstream of the network: see §D of `docs/ON-DEVICE-VERIFICATION.md`. +- **Emoji / symbols:** `tests/emoji.test.ts` + `trademarkSymbols`/`symbolsBesideEmoji` in the + round-trip `preserved` class — ©/®/™ parse as text, load into a *mounted* view, and save + byte-exact, while the emoji-presentation forms (©️ ®️ ™️) stay image emoji. Red before the fix. +- **Failed loads:** `tests/tabs.test.ts` ("an activation that throws") — a document the editor + cannot load never becomes the active tab. `onActivate` throws *after* the editor state was + replaced, so `setActive` restores and reloads the previous tab, never flushes the editor into + the failed one, and rethrows; `openPath` then drops the never-activated tab. - Plus `vaultscan`, `imgasset`, `theme`, `statusbar`, `search`, `security`, `tabs` suites. > **The browser harness.** `dev-harness.html` is `app.html` plus a fake Tauri IPC bridge diff --git a/src/editor/emoji.ts b/src/editor/emoji.ts new file mode 100644 index 0000000..59aff03 --- /dev/null +++ b/src/editor/emoji.ts @@ -0,0 +1,68 @@ +// The ONE emoji stack (CLAUDE.md §6): upstream @milkdown/plugin-emoji, plus a +// correction that keeps text twemoji does not draw as an image out of the +// `emoji` node. The app (milkdown.ts) and every gate import `emoji` from here, +// so the stack under test is the stack that ships — like canonical.ts. +// +// The defect, in @milkdown/plugin-emoji 7.21.1 and still half-present in 7.23.0: +// its remark step splits text on `emoji-regex`, which matches ©, ® and ™ (they +// are emoji with a *text* default presentation), and wraps each match in an +// `emoji` node holding `twemoji.parse(match)`. twemoji only draws the +// emoji-presentation forms (©️ with U+FE0F), so for a bare © the node holds the +// plain character. Two things then go wrong with that node: +// +// 1. It cannot be drawn. Its `toDOM` hands ProseMirror a Text node, and +// prosemirror-model ≥ 1.25.5 accepts only Element nodes in a DOM spec, so +// the load throws "Invalid array passed to renderSpec" — any note with a ® +// in it fails to open. (7.23.0 fixes this half.) +// 2. It cannot be saved. Its `toMarkdown` writes the ``'s title, and +// there is no ``, so the character is DELETED: `A © B` saves as +// `A B` (§3.2). (7.23.0 does not fix this half — upgrading alone would +// turn a visible open failure into silent loss on save.) +// +// The fix is at the source: after twemoji runs, any `emoji` node whose html is +// not an image goes back to being ordinary text. A ™ in prose is prose — it +// takes the same search, word-count, escaping and serialization path as the +// letters around it, and no custom schema is involved (§11). +import type { MilkdownPlugin } from "@milkdown/kit/ctx"; +import type { Node as MdNode } from "@milkdown/kit/transformer"; +import { $remark } from "@milkdown/kit/utils"; +import { emoji as upstreamEmoji } from "@milkdown/plugin-emoji"; + +type Literal = MdNode & { value: string }; +type Parent = MdNode & { children: MdNode[] }; + +const isParent = (node: MdNode): node is Parent => + Array.isArray((node as Partial).children); + +/** True for an `emoji` mdast node that twemoji left as plain text (no image). */ +function isTextOnlyEmoji(node: MdNode): node is Literal { + if (node.type !== "emoji") return false; + const value = (node as Partial).value; + return typeof value === "string" && !value.includes(" () => demoteTextOnlyEmoji); + +export const emoji: MilkdownPlugin[] = [...upstreamEmoji, ...remarkSymbolsAsText].flat(); diff --git a/src/editor/milkdown.ts b/src/editor/milkdown.ts index fa59e70..9d41c4b 100644 --- a/src/editor/milkdown.ts +++ b/src/editor/milkdown.ts @@ -12,7 +12,7 @@ import { listener, listenerCtx } from "@milkdown/kit/plugin/listener"; import { $prose } from "@milkdown/kit/utils"; import { EditorState, Plugin, PluginKey, TextSelection, type Selection } from "@milkdown/kit/prose/state"; import type { EditorView } from "@milkdown/kit/prose/view"; -import { emoji } from "@milkdown/plugin-emoji"; +import { emoji } from "./emoji"; import { nord } from "@milkdown/theme-nord"; import { searchPlugin } from "../ui/search"; import { useCanonical } from "./canonical"; diff --git a/src/main.ts b/src/main.ts index 02cd2e3..6e9157b 100644 --- a/src/main.ts +++ b/src/main.ts @@ -5,7 +5,7 @@ import "@milkdown/theme-nord/style.css"; import "./styles.css"; import { editorViewCtx, type Editor } from "@milkdown/kit/core"; -import { caretOf, createEditor, resetHistory } from "./editor/milkdown"; +import { type CaretSnapshot, caretOf, createEditor, resetHistory } from "./editor/milkdown"; import { docToMarkdown, markdownToDoc } from "./editor/serializer"; import { docToHtml, htmlToDoc } from "./editor/html-serializer"; import type { FrontMatter, SplitFile } from "./editor/frontmatter"; @@ -255,6 +255,21 @@ function loadIntoEditor(content: string, format: DocFormat, caret: "keep" | "sta // Taken before the swap: afterwards every position has been mapped to the end. const before = caret === "keep" ? caretOf(editor) : "start"; loading = true; + try { + loadIntoEditorUnguarded(content, format, before); + } finally { + // A load that throws (a document the view cannot draw) must not leave every + // later keystroke ignored as "still loading" — TabManager reloads the + // previous tab after the failure, and that reload is a normal load. + loading = false; + } +} + +function loadIntoEditorUnguarded( + content: string, + format: DocFormat, + before: CaretSnapshot | "start", +): void { if (format === "html") { // Front matter is a markdown-file concept; an `.html` document has none, and // the strip stays hidden for those tabs. @@ -395,7 +410,17 @@ async function openPath(path: string): Promise { return; } const file = await openFile(path); - openDocument(file.path, basename(file.path), file.content, formatForPath(file.path)); + try { + openDocument(file.path, basename(file.path), file.content, formatForPath(file.path)); + } catch (e) { + // The editor could not load it. TabManager has already put the previous tab + // back; the new tab never became active, holds exactly what is on disk, and + // can never be opened — so drop it rather than leave a dead tab whose click + // only fails again. The caller reports the error. + const failed = tabs.byPath(file.path); + if (failed && failed.id !== tabs.active()?.id && !failed.dirty) tabs.close(failed.id); + throw e; + } // Recorded from the path the backend resolved, and only once the read // succeeded — a file that failed to open must not enter a list whose whole // purpose is offering it again. @@ -403,6 +428,21 @@ async function openPath(path: string): Promise { setStatus(`Opened ${basename(file.path)}`); } +/** + * `openPath` for the fire-and-forget callers (sidebar click, a second launch, a + * drop). Awaited there, a failure was an unhandled rejection: clicking a note + * the editor could not load did nothing at all, with no message. + */ +async function openPathOrReport(path: string): Promise { + try { + await openPath(path); + return true; + } catch (e) { + setStatus(`Open failed: ${String(e)}`); + return false; + } +} + async function doOpenFile(): Promise { try { const path = await pickFileToOpen(); @@ -1995,7 +2035,7 @@ window.addEventListener("DOMContentLoaded", async () => { if (!editorRoot || !tabbar || !sidebarEl || !formatBar) return; sidebar = new Sidebar(sidebarEl, { - onOpenFile: (p) => void openPath(p), + onOpenFile: (p) => void openPathOrReport(p), // Routed through the same action as the menu item, so the empty state's // button cannot drift from File → Open Folder. onOpenFolder: () => runAction("menu_open_folder"), @@ -2145,7 +2185,7 @@ window.addEventListener("DOMContentLoaded", async () => { // While Toril is already running, a second double-click is forwarded here by // the single-instance plugin rather than starting a new process (§5). - void onOpenFile((path) => void openPath(path)); + void onOpenFile((path) => void openPathOrReport(path)); // Dropping notes on the window opens them. A drop is the one open path with // no file-type filter in front of it, so `selectOpenable` is an allowlist — @@ -2160,9 +2200,19 @@ window.addEventListener("DOMContentLoaded", async () => { return; } void (async () => { - for (const path of openable) await openPath(path); + let opened = 0; + for (const path of openable) if (await openPathOrReport(path)) opened++; const skipped = paths.length - openable.length; - if (skipped > 0) setStatus(`Opened ${openable.length}; skipped ${skipped}`); + // A failed open already put its own message up; only overwrite it with a + // summary that still mentions it. + if (skipped > 0 || opened < openable.length) { + const failed = openable.length - opened; + setStatus( + `Opened ${opened}` + + (skipped > 0 ? `; skipped ${skipped}` : "") + + (failed > 0 ? `; ${failed} failed to open` : ""), + ); + } })(); }); diff --git a/src/ui/tabs.ts b/src/ui/tabs.ts index ac551f1..16d6a35 100644 --- a/src/ui/tabs.ts +++ b/src/ui/tabs.ts @@ -142,7 +142,26 @@ export class TabManager { if (prev) this.cb.onDeactivate(prev); this.activeId = id; const next = this.active(); - if (next) this.cb.onActivate(next); + try { + if (next) this.cb.onActivate(next); + } catch (e) { + // A document the editor cannot load must never become the active tab. + // onActivate throws after the editor's *state* was replaced but before the + // screen redrew, so leaving `next` active would show the previous note + // while Save — which needs no dirty flag — wrote the half-loaded text over + // the file that failed. Put `prev` back (re-loading it is what clears the + // half-loaded state) and let the caller report the failure. The failed + // tab stays in the list with its buffer untouched: it was never active, so + // nothing was ever flushed into it, and a recovered buffer is not dropped. + this.activeId = prev?.id ?? null; + try { + if (prev) this.cb.onActivate(prev); + } catch { + this.activeId = null; // nothing loadable to fall back to + } + this.render(); + throw e; + } this.render(); } diff --git a/tests/emoji.test.ts b/tests/emoji.test.ts new file mode 100644 index 0000000..8ea7d6e --- /dev/null +++ b/tests/emoji.test.ts @@ -0,0 +1,81 @@ +// Gate for src/editor/emoji.ts: a character twemoji does not draw as an image +// (©, ®, ™ — emoji-regex matches them, twemoji leaves them as text) must stay +// ordinary text. As an upstream `emoji` node it crashed the view on load and was +// deleted on save (§3.2). Real emoji must still become emoji nodes. +import { afterEach, describe, expect, it } from "vitest"; +import { Editor, defaultValueCtx, editorViewCtx, parserCtx, rootCtx } from "@milkdown/kit/core"; +import type { Node as ProseNode } from "@milkdown/kit/prose/model"; +import { commonmark } from "@milkdown/kit/preset/commonmark"; +import { gfm } from "@milkdown/kit/preset/gfm"; +import { emoji } from "../src/editor/emoji"; +import { useCanonical } from "../src/editor/canonical"; +import { docToMarkdown, markdownToDoc } from "../src/editor/serializer"; + +let live: Editor | null = null; +afterEach(async () => { + await live?.destroy(); + live = null; + document.body.innerHTML = ""; +}); + +async function make(md = ""): Promise { + const root = document.createElement("div"); + document.body.appendChild(root); + live = await useCanonical( + Editor.make() + .config((ctx) => { + ctx.set(rootCtx, root); + ctx.set(defaultValueCtx, md); + }) + .use(commonmark) + .use(gfm) + .use(emoji), + ).create(); + return live; +} + +function emojiNodes(doc: ProseNode): string[] { + const out: string[] = []; + doc.descendants((n) => { + if (n.type.name === "emoji") out.push(String(n.attrs.html)); + }); + return out; +} + +describe("emoji: symbols twemoji leaves as text", () => { + it("parses ©, ® and ™ as text, not emoji nodes", async () => { + const editor = await make(); + const doc = editor.action((ctx) => ctx.get(parserCtx)("A © B ® C ™ D\n")); + expect(emojiNodes(doc)).toEqual([]); + expect(doc.textContent).toBe("A © B ® C ™ D"); + }); + + it("still parses a real emoji as an image emoji node", async () => { + const editor = await make(); + const doc = editor.action((ctx) => ctx.get(parserCtx)("Ship 🚀 now ™\n")); + const nodes = emojiNodes(doc); + expect(nodes).toHaveLength(1); + expect(nodes[0]).toContain(" { + const editor = await make("# Before\n"); + expect(() => markdownToDoc(editor, "Omnipaque® and Visipaque™, © 2026.\n")).not.toThrow(); + const view = editor.action((ctx) => ctx.get(editorViewCtx)); + expect(view.dom.textContent).toBe("Omnipaque® and Visipaque™, © 2026."); + expect(docToMarkdown(editor)).toBe("Omnipaque® and Visipaque™, © 2026.\n"); + }); + + // The emoji-presentation forms (U+FE0F) are what the `:tm:` shortcode and an + // emoji keyboard produce. twemoji *does* draw those, so they stay emoji nodes — + // the fix must not demote them — and they must survive a save byte-exact. + it("keeps the emoji-presentation forms ©️ ®️ ™️ as image emoji that round-trip", async () => { + const md = "Mark ©️ and ®️ and ™️ here.\n"; + const editor = await make(md); + const nodes = editor.action((ctx) => emojiNodes(ctx.get(editorViewCtx).state.doc)); + expect(nodes).toHaveLength(3); + for (const html of nodes) expect(html).toContain(" = { bracketCitation: "See figure [1] and [a, b].\n", approximateTilde: "50% stenosis, ~3 cm, 5 * 3 mm.\n", bareAutolink: "Visit https://example.com today.\n", + // emoji-regex counts ©/®/™ as emoji but twemoji leaves them as text, so + // upstream made them `emoji` nodes that crashed the view on load ("Invalid + // array passed to renderSpec") and serialized to nothing — `A © B` saved as + // `A B` (src/editor/emoji.ts). Mounting the view is part of this gate. + trademarkSymbols: "Omnipaque® and Visipaque™, © 2026.\n", + symbolsBesideEmoji: "Ship it 🚀 under the Toril™ name.\n", }; // Obsidian-specific syntax. Toril does not *render* most of it yet, but it must diff --git a/tests/tabs.test.ts b/tests/tabs.test.ts index 567f552..de95780 100644 --- a/tests/tabs.test.ts +++ b/tests/tabs.test.ts @@ -138,3 +138,71 @@ describe("TabManager", () => { expect(tab.diverged?.theirs).toBe("y\n"); // clearing one leaves the other }); }); + +// A document the editor cannot load must never become the active tab. The +// controller's onActivate throws *after* the editor's state was replaced but +// before the screen redrew, so a tab left active there is the worst of both: +// the user sees the previous note, while Save (which does not need a dirty tab) +// writes the editor's half-loaded text over the file that failed to open. +describe("TabManager: an activation that throws", () => { + function failing(badName: string) { + const container = document.createElement("div"); + const events: string[] = []; + const tm = new TabManager(container, { + onActivate: (t) => { + events.push(`activate:${t.name}`); + if (t.name === badName) throw new RangeError("cannot load"); + }, + onDeactivate: (t) => events.push(`deactivate:${t.name}`), + onCloseRequest: () => {}, + }); + return { tm, container, events }; + } + + it("keeps the previous tab active, reloads it, and rethrows", () => { + const h = failing("bad.md"); + const a = h.tm.open({ path: "/a.md", name: "a.md", content: "A" }); + h.events.length = 0; + expect(() => h.tm.open({ path: "/bad.md", name: "bad.md", content: "B" })).toThrow("cannot load"); + expect(h.tm.active()).toBe(a); + // deactivate a (flush), try bad, then reload a — the editor holds bad's + // half-loaded state until a is put back. + expect(h.events).toEqual(["deactivate:a.md", "activate:bad.md", "activate:a.md"]); + expect(h.container.querySelector('.tab[data-active="true"]')?.textContent).toContain("a.md"); + }); + + it("never flushes the editor into the tab that failed to load", () => { + const h = failing("bad.md"); + h.tm.open({ path: "/a.md", name: "a.md", content: "A" }); + try { + h.tm.open({ path: "/bad.md", name: "bad.md", content: "B" }); + } catch { + /* expected */ + } + const bad = h.tm.byPath("/bad.md"); + h.events.length = 0; + h.tm.open({ path: "/c.md", name: "c.md", content: "C" }); + // Only the tab that was really active is flushed; bad.md's buffer is untouched. + expect(h.events).toEqual(["deactivate:a.md", "activate:c.md"]); + expect(bad?.content).toBe("B"); + }); + + it("leaves no tab active when there was none to fall back to", () => { + const h = failing("bad.md"); + expect(() => h.tm.open({ path: "/bad.md", name: "bad.md", content: "B" })).toThrow(); + expect(h.tm.active()).toBeUndefined(); + }); + + it("restores the active tab when switching to an existing tab fails", () => { + const h = failing("bad.md"); + const a = h.tm.open({ path: "/a.md", name: "a.md", content: "A" }); + try { + h.tm.open({ path: "/bad.md", name: "bad.md", content: "B" }); + } catch { + /* expected */ + } + const bad = h.tm.byPath("/bad.md")!; + expect(() => h.tm.setActive(bad.id)).toThrow(); + expect(h.tm.active()).toBe(a); + }); +}); diff --git a/tests/toolbar.test.ts b/tests/toolbar.test.ts index f460932..3d274c0 100644 --- a/tests/toolbar.test.ts +++ b/tests/toolbar.test.ts @@ -18,7 +18,7 @@ import { describe, expect, it } from "vitest"; import { Editor, defaultValueCtx, editorViewCtx, rootCtx } from "@milkdown/kit/core"; import { commonmark } from "@milkdown/kit/preset/commonmark"; import { gfm } from "@milkdown/kit/preset/gfm"; -import { emoji } from "@milkdown/plugin-emoji"; +import { emoji } from "../src/editor/emoji"; import { TextSelection } from "@milkdown/kit/prose/state"; import { useCanonical } from "../src/editor/canonical"; import { docToMarkdown } from "../src/editor/serializer"; From 2280942bfea9c3bdb6363864aec68e3453b68c81 Mon Sep 17 00:00:00 2001 From: Evan Gress <106449014+evangress@users.noreply.github.com> Date: Mon, 9 Nov 2026 22:29:18 -0500 Subject: [PATCH 4/4] =?UTF-8?q?release:=20v1.1.2=20=E2=80=94=20Obsidian-sa?= =?UTF-8?q?fe=20saves=20and=20=C2=A9/=C2=AE/=E2=84=A2=20notes=20open=20aga?= =?UTF-8?q?in?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bump to 1.1.2 (package.json, tauri.conf.json, Cargo.toml/lock), point the README downloads, SECURITY.md, the issue template and the landing page at the new version, and date the CHANGELOG section that becomes the release body. The README note telling v1.1.0 Mac users to get "v1.1.1 or later" is historical and stays as written. Co-Authored-By: Claude Opus 5.5 --- .github/ISSUE_TEMPLATE/bug_report.yml | 2 +- CHANGELOG.md | 5 +++++ README.md | 24 ++++++++++++------------ SECURITY.md | 2 +- docs/index.html | 6 +++--- package.json | 2 +- src-tauri/Cargo.lock | 2 +- src-tauri/Cargo.toml | 2 +- src-tauri/tauri.conf.json | 2 +- 9 files changed, 26 insertions(+), 21 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 6528fde..e8abca2 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -42,7 +42,7 @@ body: attributes: label: Toril version description: Help → About, or the release you installed. If you built from source, give the commit. - placeholder: "v1.1.1" + placeholder: "v1.1.2" validations: required: true diff --git a/CHANGELOG.md b/CHANGELOG.md index 04598f7..bd23ada 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,11 @@ GitHub Release notes plus the commits that shipped in it. ## [Unreleased] +## [v1.1.2] — 2026-11-09 + +Saving a note in an Obsidian vault no longer breaks its links, embeds and callouts, and +notes containing ©, ® or ™ open again. + ### Fixed - **Saving a note no longer breaks Obsidian links, embeds and callouts.** When you edited a note in an Obsidian vault, saving it rewrote `[[Pneumothorax]]` as `\[\[Pneumothorax]]`, diff --git a/README.md b/README.md index 464445d..3eeec94 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ It is built on Tauri 2, TypeScript, and Milkdown. The primary platform is > **Status: 1.0.** The editor and the workflow around it work, and your notes stay > plain files you can walk away with. Installers: -> **[v1.1.1](https://github.com/kovirlabs/toril/releases/latest)**. +> **[v1.1.2](https://github.com/kovirlabs/toril/releases/latest)**. > > Keep backups of anything important, as with any editor. See > [CHANGELOG.md](./CHANGELOG.md) for what each release changed. @@ -72,18 +72,18 @@ Installers are on the | Platform | Download | |---|---| -| **Windows** | [`Toril_1.1.1_x64-setup.exe`](https://github.com/kovirlabs/toril/releases/download/v1.1.1/Toril_1.1.1_x64-setup.exe) | -| **Windows** (MSI, for IT deployment) | [`Toril_1.1.1_x64_en-US.msi`](https://github.com/kovirlabs/toril/releases/download/v1.1.1/Toril_1.1.1_x64_en-US.msi) | -| **macOS** (Apple Silicon) | [`Toril_1.1.1_aarch64.dmg`](https://github.com/kovirlabs/toril/releases/download/v1.1.1/Toril_1.1.1_aarch64.dmg) | -| **macOS** (Intel) | [`Toril_1.1.1_x64.dmg`](https://github.com/kovirlabs/toril/releases/download/v1.1.1/Toril_1.1.1_x64.dmg) | -| **Linux** (AppImage) | [`Toril_1.1.1_amd64.AppImage`](https://github.com/kovirlabs/toril/releases/download/v1.1.1/Toril_1.1.1_amd64.AppImage) | -| **Linux** (Debian/Ubuntu) | [`Toril_1.1.1_amd64.deb`](https://github.com/kovirlabs/toril/releases/download/v1.1.1/Toril_1.1.1_amd64.deb) | -| **Linux** (Fedora/RHEL) | [`Toril-1.1.1-1.x86_64.rpm`](https://github.com/kovirlabs/toril/releases/download/v1.1.1/Toril-1.1.1-1.x86_64.rpm) | +| **Windows** | [`Toril_1.1.2_x64-setup.exe`](https://github.com/kovirlabs/toril/releases/download/v1.1.2/Toril_1.1.2_x64-setup.exe) | +| **Windows** (MSI, for IT deployment) | [`Toril_1.1.2_x64_en-US.msi`](https://github.com/kovirlabs/toril/releases/download/v1.1.2/Toril_1.1.2_x64_en-US.msi) | +| **macOS** (Apple Silicon) | [`Toril_1.1.2_aarch64.dmg`](https://github.com/kovirlabs/toril/releases/download/v1.1.2/Toril_1.1.2_aarch64.dmg) | +| **macOS** (Intel) | [`Toril_1.1.2_x64.dmg`](https://github.com/kovirlabs/toril/releases/download/v1.1.2/Toril_1.1.2_x64.dmg) | +| **Linux** (AppImage) | [`Toril_1.1.2_amd64.AppImage`](https://github.com/kovirlabs/toril/releases/download/v1.1.2/Toril_1.1.2_amd64.AppImage) | +| **Linux** (Debian/Ubuntu) | [`Toril_1.1.2_amd64.deb`](https://github.com/kovirlabs/toril/releases/download/v1.1.2/Toril_1.1.2_amd64.deb) | +| **Linux** (Fedora/RHEL) | [`Toril-1.1.2-1.x86_64.rpm`](https://github.com/kovirlabs/toril/releases/download/v1.1.2/Toril-1.1.2-1.x86_64.rpm) |
Installing on Windows -**1. Download** `Toril_1.1.1_x64-setup.exe` from the table above. +**1. Download** `Toril_1.1.2_x64-setup.exe` from the table above. **2. Unblock it if Windows marked it.** Files downloaded from the internet carry a "mark of the web". If the installer refuses to start, right-click it → @@ -140,9 +140,9 @@ Toril…"*. Open **System Settings → Privacy & Security**, scroll down, and cl > later. Nothing was wrong with the download itself. **Linux:** the AppImage is portable — -`chmod +x Toril_1.1.1_amd64.AppImage` and run it. Or install the `.deb` -(`sudo apt install ./Toril_1.1.1_amd64.deb`) or `.rpm` -(`sudo dnf install ./Toril-1.1.1-1.x86_64.rpm`). +`chmod +x Toril_1.1.2_amd64.AppImage` and run it. Or install the `.deb` +(`sudo apt install ./Toril_1.1.2_amd64.deb`) or `.rpm` +(`sudo dnf install ./Toril-1.1.2-1.x86_64.rpm`).
diff --git a/SECURITY.md b/SECURITY.md index af14efa..6fbb420 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -6,7 +6,7 @@ notes are plain files on your own disk. That shapes the threat model below. ## Supported Versions -Toril is at **`v1.1.1`**. Security fixes land on the latest release and `main` +Toril is at **`v1.1.2`**. Security fixes land on the latest release and `main` only; there are no maintained back-release branches. | Version | Supported | diff --git a/docs/index.html b/docs/index.html index 3f6a196..7757fd2 100644 --- a/docs/index.html +++ b/docs/index.html @@ -47,7 +47,7 @@ "downloadUrl": "https://github.com/kovirlabs/toril/releases/latest", "installUrl": "https://github.com/kovirlabs/toril/releases/latest", "releaseNotes": "https://github.com/kovirlabs/toril/blob/main/CHANGELOG.md", - "softwareVersion": "1.1.1", + "softwareVersion": "1.1.2", "datePublished": "2026-10-15", "isAccessibleForFree": true, "license": "https://github.com/kovirlabs/toril/blob/main/LICENSE", @@ -345,7 +345,7 @@

Toril

Download View on GitHub
-
Free & open source · Apache-2.0 · v1.1.1 · Windows, macOS, Linux
+
Free & open source · Apache-2.0 · v1.1.2 · Windows, macOS, Linux
@@ -497,7 +497,7 @@

Data safety first, then features.

Download Toril

Installers for Windows (.exe, .msi), macOS (Apple Silicon and Intel) and Linux (AppImage, .deb, .rpm). Free, Apache-2.0, and nothing to sign up for.

Windows is the primary platform · Installed copies update themselves from v1.1.0 on · Changelog

diff --git a/package.json b/package.json index 72a2d1e..6cb3a54 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "toril", "private": true, - "version": "1.1.1", + "version": "1.1.2", "type": "module", "scripts": { "dev": "vite", diff --git a/src-tauri/Cargo.lock b/src-tauri/Cargo.lock index 515136d..7ac6bae 100644 --- a/src-tauri/Cargo.lock +++ b/src-tauri/Cargo.lock @@ -4841,7 +4841,7 @@ checksum = "756daf9b1013ebe47a8776667b466417e2d4c5679d441c26230efd9ef78692db" [[package]] name = "toril-app" -version = "1.1.1" +version = "1.1.2" dependencies = [ "fileops", "fsatomic", diff --git a/src-tauri/Cargo.toml b/src-tauri/Cargo.toml index f221e5b..1af8a8e 100644 --- a/src-tauri/Cargo.toml +++ b/src-tauri/Cargo.toml @@ -20,7 +20,7 @@ exclude = ["third-party/glib"] [package] name = "toril-app" -version = "1.1.1" +version = "1.1.2" description = "Toril — a WYSIWYG markdown editor" authors = ["Toril contributors"] edition = "2024" diff --git a/src-tauri/tauri.conf.json b/src-tauri/tauri.conf.json index 214c6cf..6df9668 100644 --- a/src-tauri/tauri.conf.json +++ b/src-tauri/tauri.conf.json @@ -1,7 +1,7 @@ { "$schema": "https://schema.tauri.app/config/2", "productName": "Toril", - "version": "1.1.1", + "version": "1.1.2", "identifier": "com.toril.app", "build": { "beforeDevCommand": "pnpm dev",