Live data from Hacker News

Ask HN: What is the best method for turning a scanned book as a PDF into text?

news.ycombinator.com

101–110 of 122 posts

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#101

I have tried a bunch of things. This is what worked best for me: Surya [0]. It can run fully local on your laptop. I also tried EasyOCR [1], which is also quite good. I haven't tried this myself, but I will look at Paddle [2] if the previous two don't float your boat. All of these are OSS, and you don't need to pay a dime to anyone. [0]: https://github.com/VikParuchuri/surya [1]: https://github.com/JaidedAI/EasyOCR […

Wow, Surya looks legit! https://www.datalab.to/

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#102

I have tried a bunch of things. This is what worked best for me: Surya [0]. It can run fully local on your laptop. I also tried EasyOCR [1], which is also quite good. I haven't tried this myself, but I will look at Paddle [2] if the previous two don't float your boat. All of these are OSS, and you don't need to pay a dime to anyone. [0]: https://github.com/VikParuchuri/surya [1]: https://github.com/JaidedAI/EasyOCR […

Got some questions (sorry for necro, but I only discovered this thread by accident because I left it open in a tab and it turns out to be super-relevant to me):

I have some out-of-print books that I want to convert into nice pdf's/epubs (like, reference-quality)

1) I don't mind destroying the binding to get the best quality. Any idea how I do so?

2) I have a multipage double-sided scanner (fujitsu scansnap). would this be sufficient to do the scan portion?

3) Is there anything that determines the font of the book text and reproduces that somehow? and that deals with things like bold and italic and applies that either as markdown output or what have you?

4) how do you de-paginate the raw text to reflow into (say) an epub or pdf format that will paginate based on the output device (page size/layout) specification?

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#103
I had to scan some historic books and papers for a research project and tried a couple desktop apps and python libraries. Still the Android scan to document worked better, so I just photographed all pages which was faster, then wrote an unsupervised python script to loop through the images in the camera folder and send them to Google Document AI for OCR, added some metadata and merged them. This outperformed the other even commercial solutions I tried by far and was dirt cheap. If you're interested I'll clean up the code and put it on github.

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#106

ocrmypdf with some specific prompts, depending on the source (language, force, etc.) worked for me most of the time. The biggest issue for which I have not been able to find a solution yet, is proper conversion of pdf to epub. I read a lot on my phone, and the inflexibility of pdf format, with the ugliness of "reflow" as the only apparent option to give reading the look of true epub, on phones, is frustrating.

Calibre's ebook-convert is the best PDF to X that I've found. Of course, it's not perfect but PDF is a really hard format to convert from.

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#107
Wow I asked a similar question a few days, glad to see this is getting some traction! Archive.org OCR is new to me! Very interesting. I am working on a tool to do OCR, translate and layout recovery (like .. all the apps nowadays), but focussed on running locally/processing thousands of pages/scans. Let me know if you want to collab!

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#108

Copyright issues aside (e.g. if your thing is public domain), the galaxy-brain approach is to upload your raw scanned PDF to the Internet Archive (archive.org), fill in the appropriate metadata, wait about 24 hours for their post-upload format-conversion tasks to run automatically, and then download the size-optimized and OCR-ized PDF from them. I've done this with a few documents from the French and Spanish national…

The PDFs this process creates use MRC (Mixed Raster Content), which separates each page into multiple layers: a black and white foreground layer for text/line art, a color background layer for images/colors, and a binary mask layer that controls how they're combined. This smart layering is why you can get such small file sizes while maintaining crisp text and reasonable image quality.

If you want purely black and white output (e.g. if the PDF has yellowing pages and/or not-quite-black text, but doesn't have many illustrations), you can extract just the monochrome foreground layer from each page and ignore the color layers entirely.

First, extract the images using mutool extract in.pdf

Then delete the sRGB images.

Then combine the remaining images with imagemagick command line: convert -negate *.png out.pdf

This gives you a clean black and white PDF without any of the color information or artifacts from the background layer.

Here's a script that does all that. It worked with two different PDFs from IA. I haven't tested it with other sources of MRC PDFs. The script depends on mutool and imagemagick.

https://gist.github.com/rahimnathwani/44236eaeeca10398942d2c...

Post reply on HN