Live data from Hacker News

Ask HN: What is the best method for turning a scanned book as a PDF into text?

news.ycombinator.com

31–40 of 122 posts

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#31
I have tried a bunch of things. This is what worked best for me: Surya [0]. It can run fully local on your laptop. I also tried EasyOCR [1], which is also quite good. I haven't tried this myself, but I will look at Paddle [2] if the previous two don't float your boat.

All of these are OSS, and you don't need to pay a dime to anyone.

[0]: https://github.com/VikParuchuri/surya

[1]: https://github.com/JaidedAI/EasyOCR

[2]: https://github.com/PaddlePaddle/Paddle

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#33

For years I have been printing PDFs off on regular paper and then binding them into books. 1. Print it at work when no one is looking. 2. Get two rigid boards and squeeze the stack of paper together. I customarily use two wooden armrests that originally came from a garden-furniture lounger. 3. Squeeze the paper with just a 1/4-inch showing. 4. Use wood glue and with your finger working like a toothbrush, work the glu…

It’s unclear how this is related to the article, but I’m intrigued by your simple DIY bookbinding process.

It seems straightforward except for the canvas strip (I assume this is part of the binding?), and whether you add thicker pages/boards on each side as covers.

Do you have any photos of the process, or at least of a finished product? Thanks!

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#34

I did this very recently for a 19th century book in German with occasionally some Greek. The method that produces the highest level of accuracy I've found is to use ImageMagick to extract each page as a image, then send each image file to Claude Sonnet (encoded as base64) with a simple user prompt like "Transcribe the complete text from this image verbatim with no additional commentary or explanations". The whole thi…

Is it really necessary to split it into pages? Not so bad if you automate it I suppose, but aren't there models that will accept a large PDF directly (I know Sonnet has a 32MB limit)?

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#35

I made a high-quality scan of PAIP (Paradigms of Artificial Intelligence Programming), and worked on OCR'ing and incorporating that into an admittedly imperfect git repo of Markdown files. I used Scantailor to deskew and do other adjustments before applying Tesseract, via OCRmyPDF. I wrote notes for some of my process over at https://github.com/norvig/paip-lisp/releases/tag/v1.2 . I'd also tried ocrit, which uses App…

Also this:

https://news.ycombinator.com/item?id=42952605 - Ingesting PDFs and why Gemini 2.0 changes everything

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#36
post #7

My understanding is that Gemini OCR is now considered state of the art and a material step forward in OCR accuracy

Is this from the article that was on the front page a few days ago? If so, it's not true. The title was intentionally misleading, they said they're the best, but if you read the article it was that they're actually the best in some subproblem, not the actual thing.

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#38
Copyright issues aside (e.g. if your thing is public domain), the galaxy-brain approach is to upload your raw scanned PDF to the Internet Archive (archive.org), fill in the appropriate metadata, wait about 24 hours for their post-upload format-conversion tasks to run automatically, and then download the size-optimized and OCR-ized PDF from them.

I've done this with a few documents from the French and Spanish national archives, which were originally provided as enormous non-OCRed PDFs but shrank to 10% the size (or less) after passage through archive.org and incidentally became full-text-searchable.

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#39

I did this very recently for a 19th century book in German with occasionally some Greek. The method that produces the highest level of accuracy I've found is to use ImageMagick to extract each page as a image, then send each image file to Claude Sonnet (encoded as base64) with a simple user prompt like "Transcribe the complete text from this image verbatim with no additional commentary or explanations". The whole thi…

Is it really necessary to split it into pages? Not so bad if you automate it I suppose, but aren't there models that will accept a large PDF directly (I know Sonnet has a 32MB limit)?

They are limited on how much they can output and there is generally an inverse relationship between the amount of tokens you send vs quality after the first 20-30 thousand tokens.

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#40
hey, i recommend checking out the previous HN threads [1] on why LLMs shouldn’t be used in production-grade OCR, especially if accuracy is super important (as in the audiobook case)

we wrote the post and created Pulse [2] for these exact use cases, feel free to reach out for more info!

[1]: https://news.ycombinator.com/item?id=42966958 [2]: https://runpulse.com

Post reply on HN