Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is the best method for turning a scanned book as a PDF into text?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 122 posts

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#81

Post by therealpygon »

HarHarVeryFunny wrote: post #34
aragonite wrote: post #27

I did this very recently for a 19th century book in German with occasionally some Greek. The method that produces the highest level of accuracy I've found is to use ImageMagick to extract each page as a image, then send each image file to Claude Sonnet (encoded as base64) with a simple user prompt like "Transcribe the complete text from this image verbatim with no additional commentary or explanations". The whole thi…

Is it really necessary to split it into pages? Not so bad if you automate it I suppose, but aren't there models that will accept a large PDF directly (I know Sonnet has a 32MB limit)?

Necessary? No. Better? Probably. Despite larger context windows, attention and hallucinations aren’t completely a thing of the past within the expanded context windows today. Splitting to individual pages likely helps ensure that you stay well within a normal context window size that seems to avoid most of these issues. Asking an LLM to maintain attention for a single page is much more achievable than an entire book.

Also, PDF size isn’t a relevant measurement of token lengths when it comes to PDFs which can range from a collection of high quality JPEG images to thousand(s) of pages of text

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#82

Post by brudgers »

jbaiter wrote: post #8
brudgers wrote: post #6

https://linux.die.net/man/1/pdftotext is the simplest thing that might work. It is free and mature.

That will not work for scanned PDFs without a text layer and even if it has one, it's not guaranteed to work.

"Might work" comes with neither express nor implied warranty.

OCR is another thing that might work which is also simpler than an LLM.

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#83

Post by Jedd »

knicholes wrote: post #26

I'm biased as an employee, but who knows PDFs better than Adobe? Use their PDF text extraction API.

As someone who's been using KDE's Okular PDF reader for nearly twenty years, and also has to use Adobe's products - can confidently say that at least one answer to your question is 'The developers of KDE's Okular'.

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#84

Post by aragonite »

woile wrote: post #69
aragonite wrote: post #27

I did this very recently for a 19th century book in German with occasionally some Greek. The method that produces the highest level of accuracy I've found is to use ImageMagick to extract each page as a image, then send each image file to Claude Sonnet (encoded as base64) with a simple user prompt like "Transcribe the complete text from this image verbatim with no additional commentary or explanations". The whole thi…

What about preserving the style like titles and subtitles?

You can request Markdown output, which takes care of text styling like italics and bold. For sections and subsections, in my own case they already have numerical labels (like "3.1.4") so I didn't feel the need to add extra formatting to make them stand out. Incidentally, even if you don't specify markdown output, Claude (at least in my case) automatically uses proper Unicode superscript numbers (like ¹, ², ³) for footnotes, which I find very neat.

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#86

Post by waldrews »

The smart LLM's are great at this (Gemini Flash seems accurate and cheap), but they can't be trusted not to engage in unexpected censorship, typically skipping parts they find objectionable without reliably telling you that that's what they did. That's annoying enough if you're dealing with, e.g., names that happen to spell something awkward, but it's a big problem if you're scanning medical notes or something else where the awkward text is legitimately needed.

Anyone have success with prompting them to "just give me the text verbatim?"

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#87

Post by digilypse »

waldrews wrote: post #86

The smart LLM's are great at this (Gemini Flash seems accurate and cheap), but they can't be trusted not to engage in unexpected censorship, typically skipping parts they find objectionable without reliably telling you that that's what they did. That's annoying enough if you're dealing with, e.g., names that happen to spell something awkward, but it's a big problem if you're scanning medical notes or something else w…

The API has safety configuration for this

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#88

Post by dotancohen »

KTibow wrote: post #78
dotancohen wrote: post #64

Earlier quoted context omitted.

What is the o3 model good for? Is it just an evolution of o1 (chain of thought / internal reasoning)?

Yes (albeit I believe o3-mini isn't natively multimodal)

I see, thank you.
Post reply on HN