-The Topic Modelling in the case study is performed with the [DARIAH Topics Explorer](https://dariah-de.github.io/TopicsExplorer/). It is a very easy to use tool with a graphical user interface. You can download the open source program for Windows, Mac and Linux [here](https://github.com/DARIAH-DE/TopicsExplorer/releases).
+The package
tesseract-ocr-eng installs the English language data for Tesseract, which is all you need for this lesson. If you work with sources in other languages, install the corresponding package, for example
tesseract-ocr-fra for French, or
tesseract-ocr-all for all languages. On macOS, Homebrew includes English data by default. For other languages, optionally run
brew install tesseract-lang.
-
+
-If you are using a Mac and receive an error message that the file is from an “unidentified developer,” you can overwrite it by holding control while double-clicking it. If that doesn't work, go to Systems Preferences, click on Security & Privacy, and then click Open Anyway.
+### Topic Modelling
-
-Always make a backup copy of your data before using the commands in this course. Text recognition and combining PDFs can change the original files.
+Always make a backup copy of your data before using the commands in this course. The commands in this lesson write their results to the separate folders ocr, text and images and leave the original PDF files unchanged. Never use the same file name for input and output: text recognition and combining PDFs would then overwrite your original files.
@@ -172,12 +181,21 @@ First you will get an overview of our corpus. Large databases can create a false
## Text Recognition in PDF Files
-For the text recognition, you will use [OCRmyPDF](https://ocrmypdf.readthedocs.io). This software is based on the state-of-the-art open source text recognition software [Tesseract](https://github.com/tesseract-ocr/tesseract), which is maintained and further developed by Google. The software automatically recognizes the page orientation, corrects skewed pages, cleans up image artifacts, and adds an OCR text layer to the PDF. Only the document language must be given as a parameter.
+For the text recognition, you will use [OCRmyPDF](https://ocrmypdf.readthedocs.io). This software is based on the state-of-the-art open source text recognition software [Tesseract](https://github.com/tesseract-ocr/tesseract), which was developed at Hewlett-Packard and Google and is now maintained by an open source community. The software corrects skewed pages and adds an OCR text layer to the PDF. Only the document language must be given as a parameter.
+
+First create a folder named `ocr` for the results. OCRmyPDF takes an input file and an output file. Always give it a different file for the output, so that your original PDF stays unchanged.
``` bash
-ocrmypdf --language eng --deskew --clean 'ILO-SR_N2_engl.pdf' 'ILO-SR_N2_engl.pdf'
+mkdir -p ocr
+ocrmypdf --language eng --deskew 'ILO-SR_N2_engl.pdf' 'ocr/ILO-SR_N2_engl.pdf'
```
+
+
+The optional
--clean parameter uses
unpaper to clean the image sent to text recognition. It leaves the final PDF's page images unchanged. The
--clean-final parameter also changes the final PDF's page images. Install unpaper with
sudo apt install unpaper or
brew install unpaper. Inspect the recognised text and cleaned pages, because cleaning can remove useful content.
+
+
+
+
--skip-text trusts any text that a page already contains, even if it comes from a poor earlier OCR. If a file holds a bad text layer, replace
--skip-text with
--redo-ocr to replace the OCR, or with
--force-ocr to turn every page into an image and recognise it again. When using
--redo-ocr, also remove
--deskew, because these options cannot be combined. See the
documentation for details.
+
+Before you change OCR options, move the corresponding outputs out of
ocr and
text.
+
+
@@ -242,10 +299,12 @@ For digitally created documents, Poppler extracts all embedded image files. This
-To process all PDF files in your working directory at once.
+To process all PDF files in your working directory at once:
``` bash
-find . -name '*.pdf' -exec pdfimages -png '{}' '{}' \;
+for file in *.pdf; do
+ pdfimages -png "$file" "images/$(basename "$file" .pdf)"
+done
```
Poppler can only extract illustrations if they are available as individual images in the PDF file. If you want to extract illustrations from a scanned page take a look at this lesson: [Extracting Illustrated Pages from Digital Libraries with Python](/en/lessons/extracting-illustrated-pages).
@@ -254,20 +313,28 @@ Poppler can only extract illustrations if they are available as individual image
Although OCRmyPDF can process image files directly, there are cases where you first want to combine the images into a PDF document. Because most image formats do not support multiple pages, each page of a document has to be saved as a single file. With the widespread command line image editing software [ImageMagick](https://imagemagick.org/) you can achieve this very easily.
+ImageMagick 7 uses `magick`. ImageMagick 6 uses `convert`. If you use version 6, replace `magick` with `convert` in the examples below.
+
``` bash
-convert '23B09_5_engl-002.png' '23B09_5_engl-004.png' '23B09_5_engl-006.png' '23B09_5_engl-007.png' 'some-images-combined.pdf'
+magick 'images/23B09_5_engl-002.png' 'images/23B09_5_engl-004.png' 'images/23B09_5_engl-006.png' 'images/23B09_5_engl-007.png' 'some-images-combined.pdf'
```
+
+
+Some installations prevent ImageMagick from writing PDF files. If you see a PDF security-policy error, use img2pdf instead. Install it with sudo apt install img2pdf (Ubuntu/WSL) or brew install img2pdf (macOS). Then run img2pdf images/*.png -o all-images-combined.pdf.
+
+
+
To combine all images into a PDF file at once use the wildcard operator `*.png`. This step could take a few minutes.
``` bash
-convert '*.png' 'all-images-combined.pdf'
+magick images/*.png 'all-images-combined.pdf'
```
-If you want to combine different PDF files, you can rely on Poppler. Poppler does this job much faster than ImageMagick and preserves attributes of the original documents.
+If you want to combine different PDF files, you can rely on Poppler. Poppler does this job much faster than ImageMagick and preserves attributes of the original documents. Use the files in the `ocr` folder, so that the combined PDF also contains the recognised text.
``` bash
-pdfunite 'ILO-SR_N1_engl.pdf' 'ILO-SR_N2_engl.pdf' 'ILO-SR_N3_engl.pdf' 'ILO-SR_N4_engl.pdf' 'some-pdfs-combined.pdf'
+pdfunite 'ocr/ILO-SR_N1_engl.pdf' 'ocr/ILO-SR_N2_engl.pdf' 'ocr/ILO-SR_N3_engl.pdf' 'ocr/ILO-SR_N4_engl.pdf' 'some-pdfs-combined.pdf'
```
## Use Topic Modelling to Analyze the Corpus
@@ -280,8 +347,7 @@ Now that you have performed all the steps of the PDF processing on some examples
4. Evaluate the Topic Model
- Both the download and the processing of the corpus are time consuming and resource intensive. At
doi.org/10.5281/zenodo.3582736 you can download the collection as a ZIP file and go directly to step
- 3.
+ Both the download and the processing of the corpus are time consuming and resource intensive. At
doi.org/10.5281/zenodo.3582736 you can download the extracted texts as a ZIP file. Unzip it to obtain the 340 text files, then go directly to
Create the Topic Model. Use these files wherever the instructions below refer to the
text folder. The app does not open ZIP archives.
+
+If the listing fails or its count differs from 340, check whether the ILO website has changed. Do not proceed with an incomplete corpus. Use the prepared texts from
Zenodo to continue at
Create the Topic Model.
+
+
+
+The app runs in your browser, so the size of the corpus is limited by the memory of your computer. If the app warns you that the corpus is large, check Model a sample instead to build a first model from a part of the documents. You can also lower the Max vocabulary in the Advanced settings.
+
+
### Evaluate the Topic Model
-The [DARIAH Topics Explorer](https://dariah-de.github.io/TopicsExplorer/) has a graphical user interface that makes it very easy to explore and evaluate the Topic Model and its thirty topics. In this run, the second topic looks like this (see Figure 2).
+Simple Topic Modeling has a graphical user interface that makes it very easy to explore and evaluate the Topic Model and its thirty topics. The tabs below **Run model** show the results:
+
+ - **Overview** shows all topics at a glance, a map of the topics, how prevalent each topic is in the corpus, and how similar the topics are to each other.
+ - **Topics** shows the most important words and the most representative documents of the selected topic. The words are sorted in descending order of weight and give a good overview of the topic. If you have understood a topic, give it a name with the **Rename** button.
+ - **Documents** shows which topics each document contains. You can search the texts and filter them by topic.
+ - **Diagnostics** describes the corpus and the model. These numbers help you to compare runs, but they are not a quality score.
-{% include figure.html filename="working-with-batches-of-pdf-files2.png" caption="Figure 2: DARIAH Topics Explorer showing related words, related documents and similar topics of a single topic." %}
+{% include figure.html filename="working-with-batches-of-pdf-files2.png" alt="The Topics tab in Simple Topic Modeling shows Topic 21, pension, widow and incapacity, with a bar chart of its fifteen highest-weighted terms and a count of 340 documents modelled." caption="Figure 2: Simple Topic Modeling showing the top terms of a topic about pensions and incapacity. This NMF model uses all 340 archived lesson texts, English stop words, 30 topics and random seed 42." %}
-This topic deals with various social insurance schemes. Both old-age provision and unemployment benefits are included. The words are sorted in descending order of relevance and give a good overview of the topic. You can also see which documents have the highest correspondence with this topic. As you can see when you look at related topics, this topic is close to accident insurance and legislation.
+Read the topics critically. In the run shown in Figure 2, Topic 21 includes the words ‘pension’, ‘widow’, ‘incapacity’ and ‘invalidity’, which suggest a theme of social insurance. Its representative documents include *Compensation for War Disabilities* and *The Evaluation of Permanent Incapacity for Work in Social Insurance*. Open these documents to check whether they support this interpretation. You can repeat this run with the archived corpus and the [settings used for Figure 2](/assets/working-with-batches-of-pdf-files/config.json), loaded in Step 2. Different model parameters, stop words or text recognition can change the topics, so always check their terms against the source texts.
-To further process or visualize the results with a spreadsheet program, click on the 'Export Data' button. The paper 'Parliament’s Debates about Infrastructure' by Jo Guldi illustrates how Topic Modelling can be put to use for historical research.[^4]
+To further process or visualize the results with a spreadsheet program, go to **Step 4 — Export** and click on **Download complete research package (.zip)**. The package contains the topics, their words and the topic scores of each document as CSV files, together with all charts. It also contains the file `config.json`, which records the version of the app, the stop words and all model parameters, including the random seed. Keep this file together with your results. If you load it in Step 2 later, the app restores all settings, so that you or your readers can repeat the analysis. The paper 'Parliament’s Debates about Infrastructure' by Jo Guldi illustrates how Topic Modelling can be put to use for historical research.[^4]
# Concluding Remarks
@@ -352,11 +461,11 @@ The Mueller Report is the official report documenting the findings and conclusio
This lesson focused on tools that are easy to use and are available as open source software free of charge. There are a lot of open source and commercial [alternatives](https://en.wikipedia.org/wiki/List_of_PDF_software) to process PDF files.[^7] [Getting Started with Topic Modeling and MALLET](/en/lessons/topic-modeling-and-mallet) covers one of many alternatives for Topic Modelling.
-[^1]: If you run into troubles [activating](https://docs.microsoft.com/en-us/windows/wsl/install-win10) the WSL check out the [troubleshooting](https://docs.microsoft.com/en-us/windows/wsl/troubleshooting), [documentation](https://aka.ms/wsldocs), or the [learning](https://aka.ms/learnwsl) resources.
+[^1]: If you run into troubles [installing](https://learn.microsoft.com/en-us/windows/wsl/install) the WSL check out the [troubleshooting](https://learn.microsoft.com/en-us/windows/wsl/troubleshooting) guide or the [documentation](https://learn.microsoft.com/en-us/windows/wsl/).
[^2]: In the case of a larger corpus, it is advisable to carry out random sampling instead of a detailed analysis. If no text is embedded in certain files, text recognition can be run over the entire corpus. Text recognition identifies embedded text when it is missing, and attempts to recognize it.
-[^3]: If you still want to learn more, see Ganegedara, Thushan. “Intuitive Guide to Latent Dirichlet Allocation.” *Medium*, August 23, 2018. https://towardsdatascience.com/light-on-math-machine-learning-intuitive-guide-to-latent-dirichlet-allocation-437c81220158.
+[^3]: If you still want to learn more, see Lee, Daniel D., and H. Sebastian Seung. “Learning the Parts of Objects by Non-Negative Matrix Factorization.” *Nature* 401 (1999): 788–791. https://doi.org/10.1038/44565; Blei, David M. “Probabilistic Topic Models.” *Communications of the ACM* 55, no. 4 (2012): 77–84. https://doi.org/10.1145/2133806.2133826; and Ganegedara, Thushan. “Intuitive Guide to Latent Dirichlet Allocation.” *Medium*, August 23, 2018. https://towardsdatascience.com/light-on-math-machine-learning-intuitive-guide-to-latent-dirichlet-allocation-437c81220158.
[^4]: Guldi, Jo. “Parliament’s Debates about Infrastructure: An Exercise in Using Dynamic Topic Models to Synthesize Historical Change.” *Technology and Culture* 60, no. 1 (2019): 1–33. https://doi.org/10.1353/tech.2019.0000.
diff --git a/images/working-with-batches-of-pdf-files/working-with-batches-of-pdf-files2.png b/images/working-with-batches-of-pdf-files/working-with-batches-of-pdf-files2.png
index e10b358fcf..6011cb4403 100644
Binary files a/images/working-with-batches-of-pdf-files/working-with-batches-of-pdf-files2.png and b/images/working-with-batches-of-pdf-files/working-with-batches-of-pdf-files2.png differ