Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is the best method for turning a scanned book as a PDF into text?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 122 posts

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#51

Post by maurycy »

staplung wrote: post #28
maurycy wrote: post #17

I had very good experience with `gemini-2.0-flash-exp`: https://github.com/maurycy/gemini-json-ocr

It's hard to know what to make of this because while you've included the output JSON you haven't included the input PDF so I have no idea how to interpret what it's actually doing.

Give it a try on any PDF! This is just 100 LOC, easy to audit.

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#52

Post by ant6n »

aragonite wrote: post #27

I did this very recently for a 19th century book in German with occasionally some Greek. The method that produces the highest level of accuracy I've found is to use ImageMagick to extract each page as a image, then send each image file to Claude Sonnet (encoded as base64) with a simple user prompt like "Transcribe the complete text from this image verbatim with no additional commentary or explanations". The whole thi…

I recently did some OCRing with OpenAI. I found o3-mini-hi to be imagining and changing text, whereas the older (?) o4 was more accurate. It’s a bit worrying that some of the models screw around with the text.

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#53

Post by jazzyjackson »

ant6n wrote: post #52
aragonite wrote: post #27

I did this very recently for a 19th century book in German with occasionally some Greek. The method that produces the highest level of accuracy I've found is to use ImageMagick to extract each page as a image, then send each image file to Claude Sonnet (encoded as base64) with a simple user prompt like "Transcribe the complete text from this image verbatim with no additional commentary or explanations". The whole thi…

I recently did some OCRing with OpenAI. I found o3-mini-hi to be imagining and changing text, whereas the older (?) o4 was more accurate. It’s a bit worrying that some of the models screw around with the text.

There’s GPT4, then GPT4o (o for Omni, as in multi modal) and then GPT o1 (chain of thought / internal reasoning) then o3 (because o2 is a stadium in London that I guess is very litigious about its trademark?), o3-mini is the latest but yes optimized to be faster and cheaper

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#54

Post by netfortius »

ocrmypdf with some specific prompts, depending on the source (language, force, etc.) worked for me most of the time. The biggest issue for which I have not been able to find a solution yet, is proper conversion of pdf to epub. I read a lot on my phone, and the inflexibility of pdf format, with the ugliness of "reflow" as the only apparent option to give reading the look of true epub, on phones, is frustrating.

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#55

Post by briga »

I was actually just working on a project like this to digitize an old manuscript. I used a PDF scanning app (there are plenty, I used Naps32, simple but it works). And then I piped the images into a `tesseract-ocr`. This will extract the text from the image but it won't deal with formatting or obvious typos. For that you're going to want to feed the text into an LLM with some prompt telling the model to correct errors, fix formatting, and provide clean text. Smaller local models (<70b parameters) do not work very well on this task for big documents, but I found ChatGPT's reasoning model does a fine job. My goal is to find a model that can run locally with similar performance.

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#56

Post by arctangos »

srameshc wrote: post #19

Google Cloud Document AI is amazing, I love it https://cloud.google.com/document-ai?hl=en It can correctly read many other languages than English if that is something you need. Previously I tried others and there were many errors in conversion. This does it well.

I’m curious about this api. I’d looked at it before but it didn’t seem like it could handle arbitrary input that didn’t fit one of the predefined schemas. I also wasn’t sure how much training data it needed. What has your experience been like?

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#58

Post by polshaw »

jazzyjackson wrote: post #53
ant6n wrote: post #52

Earlier quoted context omitted.

I recently did some OCRing with OpenAI. I found o3-mini-hi to be imagining and changing text, whereas the older (?) o4 was more accurate. It’s a bit worrying that some of the models screw around with the text.

There’s GPT4, then GPT4o (o for Omni, as in multi modal) and then GPT o1 (chain of thought / internal reasoning) then o3 (because o2 is a stadium in London that I guess is very litigious about its trademark?), o3-mini is the latest but yes optimized to be faster and cheaper

o2 is the UK's largest mobile network operator. They bought naming rights to what was known as the millennium dome (not even a stadium).
Post reply on HN