Live data from Hacker News

Ask HN: What is the best method for turning a scanned book as a PDF into text?

news.ycombinator.com

111–120 of 122 posts

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#111
More of a comment on your goal and something I’ve been thinking about with non-fiction books recently.

I’ve begun thinking I dislike the rigid format of non -fiction books. Few ideas bulked out and iterated, with lots of context and examples. Takes me ages to get through and I have very little free time. Cliffs notes are awful because you need some iteration and emotional connection to make the content stick.

I’d love a version of a book that has variable rates of summarization and is navigable around points and themes, so I can hop about while ensuring I don’t miss a key fact or insight buried somewhere

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#113

I have tried a bunch of things. This is what worked best for me: Surya [0]. It can run fully local on your laptop. I also tried EasyOCR [1], which is also quite good. I haven't tried this myself, but I will look at Paddle [2] if the previous two don't float your boat. All of these are OSS, and you don't need to pay a dime to anyone. [0]: https://github.com/VikParuchuri/surya [1]: https://github.com/JaidedAI/EasyOCR […

Got some questions (sorry for necro, but I only discovered this thread by accident because I left it open in a tab and it turns out to be super-relevant to me): I have some out-of-print books that I want to convert into nice pdf's/epubs (like, reference-quality) 1) I don't mind destroying the binding to get the best quality. Any idea how I do so? 2) I have a multipage double-sided scanner (fujitsu scansnap). would th…

Hey there, I don't know the answers to most of your question, honestly.

2. I think it would be enough. People do great work with much less.

3. I think Surya would handle it. I have done mostly flat text. I would also try some LLM OCR models like Google Gemini 2.0 Flash with different pipelines. With different system prompts. I am yet to do this. It would be easy to check. About fonts - never really worried about it myself. If it's something fancy, and you are crazy enough, you will create a font. Or you can also use some handwriting mimicry tool using another AI model. I don't have a name on top of my head. Look through OCR models. Indian college and HS kids still have to submit handwritten projects and assignments. Some crafty kids use such tools to type (or chatgpt copy-paste) and then print in pen ink color in their own handwriting, and fool the teacher given there are a large number of assignments to check.

4. I am not sure if I understand the question fully. Do you mean that books' pages will have numbers, and they will be read as book text in your OCRed data? If you mean that, then I just used GOF regex to root page numbers out. When you have the full text without page numbers, there are multiple tools to create EPUBs and PDF's. You can also reformat documents, assuming you already have an EPUB or PDF- based on the target device, using just Calibre.

1. I don't understand the question. You mean any other kind of scan than regular scanning? I don't know at all. I just work with regularly scanned documents.

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#114

More of a comment on your goal and something I’ve been thinking about with non-fiction books recently. I’ve begun thinking I dislike the rigid format of non -fiction books. Few ideas bulked out and iterated, with lots of context and examples. Takes me ages to get through and I have very little free time. Cliffs notes are awful because you need some iteration and emotional connection to make the content stick. I’d lov…

There are strategies you can employ for this. Many are spelled out in an HN fave, How to Read a Book (1940), by Mortimer J. Adler (https://archive.org/details/howtoreadabook1940edition>, Wikipedia: https://en.wikipedia.org/wiki/How_to_Read_a_Book>). It describes both types of books (fiction, instructive, and others), levels of reading, and specific reading strategies.

You can also pre-digest many such books in audio form (increasingly using what are now fairly powerful and tolerable text-to-speech tools), and dive in to read specific passages of note.

Because there's a formula to the book structure, you'll often find theory/overview / solutions presented in the introductory and concluding chapters or sections, with the mid-bulk section largely consisting of illustrations. There's an exceptionally ill-conceived notion that 1) one must finish all books one begins and 2) one must read all of a book. Neither of these are true, and I find books most usefully engaged as conversations with an author (books are conversations over time, telecoms are communications over space), and to read with a view to addressing specific goals: understanding of specific topics / problems / solutions, etc. This can cut your overall interaction with tedious works.

There's also of course a huge marketing dynamic to book publishing on which one of the best treatments is Arthur Schopenhauer's "On Authorship": (trans. 1897)

... Writing for money and reservation of copyright are, at bottom, the ruin of literature. No one writes anything that is worth writing, unless he writes entirely for the sake of his subject. What an inestimable boon it would be, if in every branch of literature there were only a few books, but those excellent! This can never happen, as long as money is to be made by writing. It seems as though the money lay under a curse; for every author degenerates as soon as he begins to put pen to paper in any way for the sake of gain. The best works of the greatest men all come from the time when they had to write for nothing or for very little. And here, too, that Spanish proverb holds good, which declares that honor and money are not to be found in the same purse—honora y provecho no caben en un saco. The reason why Literature is in such a bad plight nowadays is simply and solely that people write books to make money. A man who is in want sits down and writes a book, and the public is stupid enough to buy it. The secondary effect of this is the ruin of language. ...

https://en.wikisource.org/wiki/The_Art_of_Literature/On_Auth...>

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#115
post #44

Earlier quoted context omitted.

Last time I checked a few months ago, LLMs were more accurate than the OCR that the archive is using. The web archive version is/was not using context to figure out that for example “in the garden was a trge” should be “in the garden was a tree”. LLMs depending on the prompt do this.

Perhaps. My perhaps-curmudgeonly take on that is that it sounds a bit like "Xerox scanners/photocopiers randomly alter numbers in scanned documents" ( https://news.ycombinator.com/item?id=29223815 ). I'd much rather deal with "In the garden was a trge" than "In the garden was a tree," for example, if what the page actually said was "In the garden was a tiger." That said, of course you're right that context is useful…

Fair enough. Very valid points. I guess it boils down to “test both systems and see what works best for the task at hand”. I can indeed imagine cases were your approach would be the better option for sure.

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#117

I did this very recently for a 19th century book in German with occasionally some Greek. The method that produces the highest level of accuracy I've found is to use ImageMagick to extract each page as a image, then send each image file to Claude Sonnet (encoded as base64) with a simple user prompt like "Transcribe the complete text from this image verbatim with no additional commentary or explanations". The whole thi…

how big were the image files in terms of size/resolution that go you the level of accuracy you needed with Claude?

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#118
post #117

I did this very recently for a 19th century book in German with occasionally some Greek. The method that produces the highest level of accuracy I've found is to use ImageMagick to extract each page as a image, then send each image file to Claude Sonnet (encoded as base64) with a simple user prompt like "Transcribe the complete text from this image verbatim with no additional commentary or explanations". The whole thi…

how big were the image files in terms of size/resolution that go you the level of accuracy you needed with Claude?

300dpi (`magick -density 300 book.pdf page_%03d.png` was the command I used). The PDF is a from archieve.org & a very high-quality scan (https://ia601307.us.archive.org/5/items/derlgnertheori00rsuo...)

Re: Ask HN: What is the best method for turning a scanned book as a PDF into text?

#119
post #96

I did this very recently for a 19th century book in German with occasionally some Greek. The method that produces the highest level of accuracy I've found is to use ImageMagick to extract each page as a image, then send each image file to Claude Sonnet (encoded as base64) with a simple user prompt like "Transcribe the complete text from this image verbatim with no additional commentary or explanations". The whole thi…

Do you have a rough estimate of what the price per page was for this?

It must have been under $3 for the 150 or so API calls, possibly even under $2, though I'm less sure about that.
Post reply on HN