From 11149a52842ad16d48f8678109ff63d183533013 Mon Sep 17 00:00:00 2001 From: Ridge Kimani Date: Thu, 8 Oct 2026 14:07:18 +0300 Subject: [PATCH 1/4] fix: render math as native MathML at scrape time --- Gemfile | 1 + Gemfile.lock | 4 + README.md | 1 + assets/stylesheets/global/_base.scss | 3 + docs/filter-reference.md | 1 + lib/docs/core/math_renderer.rb | 94 +++++++++++ lib/docs/core/scraper.rb | 2 +- lib/docs/filters/core/clean_text.rb | 2 +- lib/docs/filters/core/math.rb | 130 ++++++++++++++ lib/docs/filters/pytorch/clean_html.rb | 2 - test/lib/docs/core/math_renderer_test.rb | 74 ++++++++ test/lib/docs/filters/core/clean_text_test.rb | 5 + test/lib/docs/filters/core/math_test.rb | 158 ++++++++++++++++++ 13 files changed, 473 insertions(+), 4 deletions(-) create mode 100644 lib/docs/core/math_renderer.rb create mode 100644 lib/docs/filters/core/math.rb create mode 100644 test/lib/docs/core/math_renderer_test.rb create mode 100644 test/lib/docs/filters/core/math_test.rb diff --git a/Gemfile b/Gemfile index 526f50621b..997d90514a 100644 --- a/Gemfile +++ b/Gemfile @@ -35,6 +35,7 @@ end group :docs do gem 'image_optim_pack', platforms: :ruby gem 'image_optim' + gem 'katex', '~> 0.11', require: false gem 'kramdown' gem 'kramdown-parser-gfm' gem 'redcarpet' diff --git a/Gemfile.lock b/Gemfile.lock index 20eea3352a..8969dc2187 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -134,6 +134,8 @@ GEM rdoc (>= 4.0.0) reline (>= 0.4.2) json (2.19.4) + katex (0.11.0) + execjs (~> 2.8) kramdown (2.5.2) rexml (>= 3.4.4) kramdown-parser-gfm (1.1.0) @@ -315,6 +317,7 @@ DEPENDENCIES html-pipeline (~> 2.14) image_optim image_optim_pack + katex (~> 0.11) kramdown kramdown-parser-gfm minitest @@ -396,6 +399,7 @@ CHECKSUMS io-console (0.8.2) sha256=d6e3ae7a7cc7574f4b8893b4fca2162e57a825b223a177b7afa236c5ef9814cc irb (1.18.0) sha256=de9454a0703a54704b9811a5ef31a60c86949fbf4013fcf244fabc7c775248e3 json (2.19.4) sha256=670a7d333fb3b18ca5b29cb255eb7bef099e40d88c02c80bd42a3f30fe5239ac + katex (0.11.0) sha256=6f9fa9a13e2ba616a0f8d783cc9b30979f0eefb0f5f1e04e66fe7dd8eb682e78 kramdown (2.5.2) sha256=1ba542204c66b6f9111ff00dcc26075b95b220b07f2905d8261740c82f7f02fa kramdown-parser-gfm (1.1.0) sha256=fb39745516427d2988543bf01fc4cf0ab1149476382393e0e9c48592f6581729 logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203 diff --git a/README.md b/README.md index 2ab43c29da..d57d8b6a35 100644 --- a/README.md +++ b/README.md @@ -114,6 +114,7 @@ Modifications made to each document include: * replacing all internal (scraped) URLs with their unqualified and relative counterpart * adding content, such as a title and link to the original document * ensuring correct syntax highlighting using [Prism](http://prismjs.com/) +* converting math markup to native MathML using [KaTeX](https://katex.org/) These modifications are applied via a set of filters using the [HTML::Pipeline](https://github.com/jch/html-pipeline) library. Each scraper includes filters specific to itself, one of which is tasked with figuring out the pages' metadata. diff --git a/assets/stylesheets/global/_base.scss b/assets/stylesheets/global/_base.scss index 640ca929a5..527d631ff9 100644 --- a/assets/stylesheets/global/_base.scss +++ b/assets/stylesheets/global/_base.scss @@ -149,6 +149,9 @@ table { max-width: 100%; } +// No overflow rule here: a scroll container on a element leaves it blank in Chrome. +math[display="block"] { margin: 1.5em 0; } + caption { font-weight: var(--boldFontWeight); padding: 0 .7em .3em; diff --git a/docs/filter-reference.md b/docs/filter-reference.md index a98f1778a1..a9d9b8916a 100644 --- a/docs/filter-reference.md +++ b/docs/filter-reference.md @@ -74,6 +74,7 @@ The `call` method must return either `doc` or `html`, depending on the type of f ## Core filters * [`ContainerFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside) +* [`MathFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/math.rb) — converts math markup (Sphinx/MathJax `\(...\)` and `\[...\]`, MathJax `\(...\) and
\[...\]
+ # - MathJax 2: and type="math/tex; mode=display" + # - MathJax 3: (keeps its assistive MathML, if any) + # - KaTeX: pre-rendered nodes (re-rendered from their TeX source) + # - Dollar delimiters ($...$ and $$...$$) in text, only when context[:math_dollars] is true + # Existing elements are left untouched. + # + # This filter must run before CleanHtmlFilter, which removes 
 for invalid display TeX" do
+      assert_equal '
\frac{a
', renderer.render('\frac{a', display: true)
+    end
+  end
+end
diff --git a/test/lib/docs/filters/core/clean_text_test.rb b/test/lib/docs/filters/core/clean_text_test.rb
index 4126b88686..630b754e4e 100644
--- a/test/lib/docs/filters/core/clean_text_test.rb
+++ b/test/lib/docs/filters/core/clean_text_test.rb
@@ -15,6 +15,11 @@ class CleanTextFilterTest < Minitest::Spec
     assert_equal @body, filter_output
   end
 
+  it "doesn't remove empty MathML nodes" do
+    @body = " "
+    assert_equal @body, filter_output
+  end
+
   it "strips leading and trailing whitespace" do
     @body = "\n\r Test \r\n"
     assert_equal 'Test', filter_output
diff --git a/test/lib/docs/filters/core/math_test.rb b/test/lib/docs/filters/core/math_test.rb
new file mode 100644
index 0000000000..eae614d6eb
--- /dev/null
+++ b/test/lib/docs/filters/core/math_test.rb
@@ -0,0 +1,158 @@
+require_relative '../../../../test_helper'
+require_relative '../../../../../lib/docs'
+
+class MathFilterTest < Minitest::Spec
+  include FilterTestHelper
+  self.filter_class = Docs::MathFilter
+  self.filter_type = 'html'
+
+  def math_nodes
+    filter_output.css('math')
+  end
+
+  def tex_of(node)
+    node.at_css('annotation[encoding="application/x-tex"]').content
+  end
+
+  it "runs before CleanHtmlFilter so that MathJax 
'
+      assert_equal 1, math_nodes.length
+      assert_equal 'x < y', tex_of(math_nodes.first)
+      assert_nil math_nodes.first['display']
+      assert_empty filter_output.css('script, .MathJax_Preview')
+    end
+
+    it "replaces '
+      assert_equal 'block', math_nodes.first['display']
+    end
+  end
+
+  context "MathJax 3 markup" do
+    it "replaces  with its assistive MathML" do
+      @body = '
x
'
+      assert_equal '
x
', filter_output_string
+    end
+
+    it "leaves  without assistive MathML untouched" do
+      @body = ''
+      assert_equal @body, filter_output_string
+    end
+  end
+
+  context "KaTeX markup" do
+    def katex(tex, mathml: true)
+      annotation = tex ? %(#{tex}) : ''
+      math = mathml ? %(z#{annotation}) : ''
+      %(#{math})
+    end
+
+    it "re-renders the TeX source of inline math" do
+      @body = %(
#{katex('y = xA^T + b')}
)
+      assert_equal 1, math_nodes.length
+      assert_equal 'y = xA^T + b', tex_of(math_nodes.first)
+      assert_includes math_nodes.first.to_s, 'y'
+      assert_empty filter_output.css('.katex, .katex-html, .katex-mathml')
+    end
+
+    it "re-renders the TeX source of display math" do
+      @body = %(#{katex('y')})
+      assert_equal 'block', math_nodes.first['display']
+      assert_empty filter_output.css('.katex-display')
+    end
+
+    it "keeps the existing MathML when there's no TeX source" do
+      @body = %(
#{katex(nil)}
)
+      assert_equal '
z
', filter_output_string
+    end
+
+    it "leaves nodes without MathML or TeX source untouched" do
+      @body = katex(nil, mathml: false)
+      assert_equal @body, filter_output_string
+    end
+  end
+
+  context "dollar delimiters" do
+    before { @body = '
Pay $5 and $10 for $x$ and $$y$$, not $x$.
' }
+
+    it "are ignored by default" do
+      assert_equal @body, filter_output_string
+    end
+
+    context "when context[:math_dollars] is true" do
+      before { context[:math_dollars] = true }
+
+      it "are rendered" do
+        assert_equal %w(x y), math_nodes.map { |node| tex_of(node) }
+        assert_equal 'block', math_nodes.last['display']
+      end
+
+      it "aren't mistaken for currency" do
+        assert_includes filter_output_string, 'Pay $5 and $10 for '
+      end
+
+      it "aren't rendered inside " do
+        assert_equal '$x$', filter_output.at_css('code').to_s
+      end
+    end
+  end
+end

From 9c97c92eb30a0a6e21576721e6f86a5518734acc Mon Sep 17 00:00:00 2001
From: Simon Legner 
Date: Thu, 15 Oct 2026 17:13:51 +0200
Subject: [PATCH 2/4] math: handle Sphinx markup only

Drop the MathJax 2/3, KaTeX and $-delimiter paths: no scraper needs
them, and PyTorch's KaTeX output already ships MathML.
---
 docs/filter-reference.md                |   2 +-
 lib/docs/filters/core/math.rb           | 122 +++++-------------------
 lib/docs/filters/pytorch/clean_html.rb  |   2 +
 test/lib/docs/filters/core/math_test.rb |  91 +-----------------
 4 files changed, 28 insertions(+), 189 deletions(-)

diff --git a/docs/filter-reference.md b/docs/filter-reference.md
index a9d9b8916a..00139d524b 100644
--- a/docs/filter-reference.md
+++ b/docs/filter-reference.md
@@ -74,7 +74,7 @@ The `call` method must return either `doc` or `html`, depending on the type of f
 ## Core filters
 
 * [`ContainerFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside)
-* [`MathFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/math.rb) — converts math markup (Sphinx/MathJax `\(...\)` and `\[...\]`, MathJax ` and type="math/tex; mode=display"
-  # - MathJax 3:  (keeps its assistive MathML, if any)
-  # - KaTeX: pre-rendered  nodes (re-rendered from their TeX source)
-  # - Dollar delimiters ($...$ and $$...$$) in text, only when context[:math_dollars] is true
-  # Existing  elements are left untouched.
-  #
-  # This filter must run before CleanHtmlFilter, which removes \(...\) and 
\[...\]

   class MathFilter < Filter
     TEX_DELIMITERS_RGX = /\\\((.*?)\\\)|\\\[(.*?)\\\]/m
-    DOLLARS_RGX = /\$\$(?[^$]+?)\$\$|(?[^\s$](?:[^$\n]*?[^\s$\\])?)\$(?![\d$])/
-    TEX_ANNOTATION = 'annotation[encoding="application/x-tex"]'
 
     def call
-      @items = []
-      extract_sphinx
-      extract_mathjax2
-      extract_mathjax3
-      extract_katex
-      extract_dollars if context[:math_dollars]
-      render
-      doc
-    end
-
-    private
-
-    def extract_sphinx
-      css('span.math', 'div.math').each do |node|
+      items = css('span.math', 'div.math').filter_map do |node|
         node.css('.eqno').remove
         content = node.content
         next unless content =~ TEX_DELIMITERS_RGX
-
-        pieces = []
-        last = 0
-        content.scan(TEX_DELIMITERS_RGX) do
-          match = Regexp.last_match
-          pieces << match.pre_match[last..] unless match.begin(0) == last
-          pieces << [match[1] || match[2], !match[1]]
-          last = match.end(0)
-        end
-        pieces << content[last..] if last < content.length
-
-        queue(node, pieces, id: node['id'])
-      end
-    end
-
-    def extract_mathjax2
-      css('.MathJax_Preview').remove
-      css('script[type^="math/tex"]').each do |node|
-        display = node['type'].include?('mode=display')
-        queue(node, [[node.content, display]])
-      end
-    end
-
-    def extract_mathjax3
-      css('mjx-container').each do |node|
-        math = node.at_css('mjx-assistive-mml > math')
-        next unless math
-        math['display'] = 'block' if node['display'] == 'true'
-        node.replace(math)
-      end
-    end
-
-    def extract_katex
-      css('.katex').each do |node|
-        wrapper = node.parent if node.parent.try(:[], 'class').to_s.split.include?('katex-display')
-        target = wrapper || node
-        annotation = node.at_css(TEX_ANNOTATION)
-        math = node.at_css('math')
-        display = !wrapper.nil? || (math && math['display'] == 'block')
-
-        if annotation
-          queue(target, [[annotation.content, display]])
-        elsif math
-          target.replace(math)
-        end
+        [node, split(content)]
       end
+      render(items)
+      doc
     end
 
-    def extract_dollars
-      xpath('.//text()[not(ancestor::pre or ancestor::code or ancestor::script or ancestor::style or ancestor::math)]').each do |node|
-        content = node.content
-        next unless content =~ DOLLARS_RGX
-
-        pieces = []
-        last = 0
-        content.scan(DOLLARS_RGX) do
-          match = Regexp.last_match
-          pieces << match.pre_match[last..] unless match.begin(0) == last
-          pieces << [match[:display] || match[:inline], !match[:display].nil?]
-          last = match.end(0)
-        end
-        pieces << content[last..] if last < content.length
+    private
 
-        queue(node, pieces)
+    # Splits text into plain strings and [tex, display] pairs.
+    def split(content)
+      pieces = []
+      last = 0
+      content.scan(TEX_DELIMITERS_RGX) do
+        match = Regexp.last_match
+        pieces << content[last...match.begin(0)] unless match.begin(0) == last
+        pieces << [match[1] || match[2], !match[1]]
+        last = match.end(0)
       end
+      pieces << content[last..] if last < content.length
+      pieces
     end
 
-    def queue(node, pieces, id: nil)
-      @items << [node, pieces, id]
-    end
-
-    def render
-      expressions = @items.flat_map { |_, pieces, _| pieces.grep(Array) }
+    def render(items)
+      expressions = items.flat_map { |_, pieces| pieces.grep(Array) }
       return if expressions.empty?
 
       rendered = MathRenderer.render_all(expressions)
 
-      @items.each do |node, pieces, id|
+      items.each do |node, pieces|
         html = pieces.map do |piece|
           next CGI.escapeHTML(piece) if piece.is_a?(String)
           tex, display = piece
-          result = rendered.shift
-          result[:mathml] || MathRenderer.fallback(tex, display: display)
+          rendered.shift[:mathml] || MathRenderer.fallback(tex, display: display)
         end.join
 
         fragment = doc.document.fragment(html)
-        fragment.at_css('math')['id'] = id if id && fragment.at_css('math')
+        fragment.at_css('math')['id'] = node['id'] if node['id'] && fragment.at_css('math')
         node.replace(fragment)
       end
     end
diff --git a/lib/docs/filters/pytorch/clean_html.rb b/lib/docs/filters/pytorch/clean_html.rb
index 1cca80f078..455b939ee4 100644
--- a/lib/docs/filters/pytorch/clean_html.rb
+++ b/lib/docs/filters/pytorch/clean_html.rb
@@ -4,6 +4,8 @@ class CleanHtmlFilter < Filter
       def call
         if root = at_css('#pytorch-article')
           @doc = root
+          # Show katex-mathml nodes and remove katex-html nodes
+          css('.katex-html').remove
         end
         doc
       end
diff --git a/test/lib/docs/filters/core/math_test.rb b/test/lib/docs/filters/core/math_test.rb
index eae614d6eb..d5b1f6d086 100644
--- a/test/lib/docs/filters/core/math_test.rb
+++ b/test/lib/docs/filters/core/math_test.rb
@@ -14,13 +14,8 @@ def tex_of(node)
     node.at_css('annotation[encoding="application/x-tex"]').content
   end
 
-  it "runs before CleanHtmlFilter so that MathJax 
'
-      assert_equal 1, math_nodes.length
-      assert_equal 'x < y', tex_of(math_nodes.first)
-      assert_nil math_nodes.first['display']
-      assert_empty filter_output.css('script, .MathJax_Preview')
-    end
-
-    it "replaces '
-      assert_equal 'block', math_nodes.first['display']
-    end
-  end
-
-  context "MathJax 3 markup" do
-    it "replaces  with its assistive MathML" do
-      @body = '
x
'
-      assert_equal '
x
', filter_output_string
-    end
-
-    it "leaves  without assistive MathML untouched" do
-      @body = ''
-      assert_equal @body, filter_output_string
-    end
-  end
-
-  context "KaTeX markup" do
-    def katex(tex, mathml: true)
-      annotation = tex ? %(#{tex}) : ''
-      math = mathml ? %(z#{annotation}) : ''
-      %(#{math})
-    end
-
-    it "re-renders the TeX source of inline math" do
-      @body = %(
#{katex('y = xA^T + b')}
)
-      assert_equal 1, math_nodes.length
-      assert_equal 'y = xA^T + b', tex_of(math_nodes.first)
-      assert_includes math_nodes.first.to_s, 'y'
-      assert_empty filter_output.css('.katex, .katex-html, .katex-mathml')
-    end
-
-    it "re-renders the TeX source of display math" do
-      @body = %(#{katex('y')})
-      assert_equal 'block', math_nodes.first['display']
-      assert_empty filter_output.css('.katex-display')
-    end
-
-    it "keeps the existing MathML when there's no TeX source" do
-      @body = %(
#{katex(nil)}
)
-      assert_equal '
z
', filter_output_string
-    end
-
-    it "leaves nodes without MathML or TeX source untouched" do
-      @body = katex(nil, mathml: false)
-      assert_equal @body, filter_output_string
-    end
-  end
-
-  context "dollar delimiters" do
-    before { @body = '
Pay $5 and $10 for $x$ and $$y$$, not $x$.
' }
-
-    it "are ignored by default" do
-      assert_equal @body, filter_output_string
-    end
-
-    context "when context[:math_dollars] is true" do
-      before { context[:math_dollars] = true }
-
-      it "are rendered" do
-        assert_equal %w(x y), math_nodes.map { |node| tex_of(node) }
-        assert_equal 'block', math_nodes.last['display']
-      end
-
-      it "aren't mistaken for currency" do
-        assert_includes filter_output_string, 'Pay $5 and $10 for '
-      end
-
-      it "aren't rendered inside " do
-        assert_equal '$x$', filter_output.at_css('code').to_s
-      end
-    end
-  end
 end

From ece781cacf1f4762cbb312894e938582080f225e Mon Sep 17 00:00:00 2001
From: Simon Legner 
Date: Thu, 15 Oct 2026 17:14:15 +0200
Subject: [PATCH 3/4] math: make the filter opt-in

Remove it from the default filter stack and enable it for the Sphinx
scrapers whose pages contain math: NumPy, scikit-learn, scikit-image,
Polars and statsmodels.
---
 docs/filter-reference.md          | 2 +-
 lib/docs/core/scraper.rb          | 2 +-
 lib/docs/scrapers/numpy.rb        | 2 +-
 lib/docs/scrapers/polars.rb       | 2 +-
 lib/docs/scrapers/scikit_image.rb | 2 +-
 lib/docs/scrapers/scikit_learn.rb | 2 +-
 lib/docs/scrapers/statsmodels.rb  | 2 +-
 7 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/docs/filter-reference.md b/docs/filter-reference.md
index 00139d524b..04114fa600 100644
--- a/docs/filter-reference.md
+++ b/docs/filter-reference.md
@@ -74,7 +74,7 @@ The `call` method must return either `doc` or `html`, depending on the type of f
 ## Core filters
 
 * [`ContainerFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside)
-* [`MathFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/math.rb) — converts Sphinx/MathJax math markup (`\(...\)` and `\[...\]`) to native MathML using [KaTeX](https://katex.org/), which requires a JavaScript runtime such as Node.js
+* [`MathFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/math.rb) — converts Sphinx/MathJax math markup (`\(...\)` and `\[...\]`) to native MathML using [KaTeX](https://katex.org/), which requires a JavaScript runtime such as Node.js; not in the default stack, add `math` to a scraper's `html_filters` to use it
 * [`CleanHtmlFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/clean_html.rb) — removes HTML comments, `