Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

github.com

Post reply on HN
Print view View original

71–80 of 178 posts

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#71

Post by kbyatnal »

"real improvements came from adjusting the prompts to make things clearer for the model, and not asking the model to do too much in a single pass"

This is spot on, and it's the same as how humans behave. If you give a human too many instructions at once, they won't follow all of them accurately.

I spend a lot of time thinking about LLMs + documents, and in my opinion, as the models get better, OCR is soon going to be a fully solved problem. The challenge then becomes explaining the ambiguity and intricacies of complex documents to AI models in an effective way, less so about the OCR capabilities itself.

disclaimer: I run a LLM document processing company called Extend (https://www.extend.app/).

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#73

Post by eigenvalue »

esafak wrote: post #72

I'd suggest measuring the word- and character error rates with and without the LLM. It'll let people quickly know how well it works.

Not sure how you would do that without having the ground truth to compare to. It's also very hard to measure once you start messing with the formatting (like converting it to markdown or suppressing page numbers and repeated headers/footers). I think it would also vary a lot depending on the quality of the original scan and the format and content of the document. There's really no substitute from just trying it on your document and then quickly looking through the output by hand (at least for now-- probably in a year models we be good enough and have big enough context windows to do this really well, too!).

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#74

Post by eigenvalue »

savikko wrote: post #42
eigenvalue wrote: post #5

Earlier quoted context omitted.

You know, I’ve really looked hard at what’s out there and haven’t been able to find anything else that’s totally free/open, that runs well on CPU, and which has better quality output than Tesseract. I found a couple Chinese projects but had trouble getting them to work and the documentation wasn’t great. If you have any leads on others to try I’d love to hear about them. One of the benefits of this project is that it…

I have some pretty good experiences with PaddleOCR but you may refer to this Chinese and badly documented ones. For our use case PaddleOCR + LLM has been quite nice combo.

Yes, that's one of the ones I tried. It seemed to be more designed for things like receipts and menus rather than books. But in any case, I found it hard to set up and use (and it's likely slow on the CPU compared to Tesseract, which despite its low accuracy, is at least very fast on CPU).

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#75

Post by vikp »

aliosm wrote: post #65

I'm working on Arabic OCR for a massive collection of books and pages (over 13 million pages so far). I've tried multiple open-source models and projects, including Tesseract, Surya, and a Nougat small model fine-tuned for Arabic. However, none of them matched the latency and accuracy of Google OCR. As a result, I developed a Python package called tahweel ( https://github.com/ieasybooks/tahweel ), which leverages Goo…

Hi, I'm the author of surya (https://github.com/VikParuchuri/surya) - working on improving speed and accuracy now. Happy to collaborate if you have specific page types it's not working on. For modern/clean documents it benchmarks very similarly to Google Cloud, but working on supporting older documents better now.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#76

Post by troysk »

ChadNauseam wrote: post #70
troysk wrote: post #63

In my experience, this works well but doesn't scale to all kinds of documents. For scientific papers; it can't render formulas. meta's nougat is the best model to do that. For invoices and records; donut works better. Both these models will fail in some cases so you end up running LLM to fix the issues. Even with that LLM won't be able to do tables and charts justice, as the details were lost during OCR process (bold…

It's not OSS, but I've had good experiences with using MathPix's API for OCR for formulas

nougat, donut are OSS. There are no OSS vision models but we will soon have them. MathPix API are also not OSS and I found them expensive compared to vision models.

Mathpix Markdown however is awesome and I ask LLMs to use that to denote formulas as latex is tricky to render in HTML because of things not matching. I don't know latex well so haven't gone deeper on it.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#77

Post by esafak »

eigenvalue wrote: post #73
esafak wrote: post #72

I'd suggest measuring the word- and character error rates with and without the LLM. It'll let people quickly know how well it works.

Not sure how you would do that without having the ground truth to compare to. It's also very hard to measure once you start messing with the formatting (like converting it to markdown or suppressing page numbers and repeated headers/footers). I think it would also vary a lot depending on the quality of the original scan and the format and content of the document. There's really no substitute from just trying it on yo…

You can correct the transcript to create the ground truth. Or print your own document, then run OCR on it.

OCR evaluation has been a thing for decades.

edit: Better than a single document, process a standard OCR dataset: https://paperswithcode.com/task/optical-character-recognitio...

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#78

Post by vintermann »

troysk wrote: post #63

In my experience, this works well but doesn't scale to all kinds of documents. For scientific papers; it can't render formulas. meta's nougat is the best model to do that. For invoices and records; donut works better. Both these models will fail in some cases so you end up running LLM to fix the issues. Even with that LLM won't be able to do tables and charts justice, as the details were lost during OCR process (bold…

I agree that vision models that actually have access to the image are a more sound approach than using OCR and trying to fix it up. It may be more expensive though, and depending on what you're trying to do it may be good enough.

What I want to do is reading handwritten documents from the 18th century, and I feel like the multistep approach hits a hard ceiling there. Transkribus is multistep, but the line detecion model is just terrible. Things that should be easy, such as printed schemas, utterly confuse it. You simply need to be smart about context to a much higher degree than you need in OCR of typewriter-written text.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#79

Post by bugglebeetle »

aliosm wrote: post #65

I'm working on Arabic OCR for a massive collection of books and pages (over 13 million pages so far). I've tried multiple open-source models and projects, including Tesseract, Surya, and a Nougat small model fine-tuned for Arabic. However, none of them matched the latency and accuracy of Google OCR. As a result, I developed a Python package called tahweel ( https://github.com/ieasybooks/tahweel ), which leverages Goo…

This has been my experience with Japanese texts as well. I have a number of fairly obscure Japanese books and magazines I’ve collected as part of a research interest. During the pandemic, I began digitizing them and found that nothing but Google OCR could extract the text correctly. I recently tried again with the libraries you mentioned, but they also performed worse than traditional tools.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#80

Post by aliosm »

vikp wrote: post #75
aliosm wrote: post #65

I'm working on Arabic OCR for a massive collection of books and pages (over 13 million pages so far). I've tried multiple open-source models and projects, including Tesseract, Surya, and a Nougat small model fine-tuned for Arabic. However, none of them matched the latency and accuracy of Google OCR. As a result, I developed a Python package called tahweel ( https://github.com/ieasybooks/tahweel ), which leverages Goo…

Hi, I'm the author of surya ( https://github.com/VikParuchuri/surya ) - working on improving speed and accuracy now. Happy to collaborate if you have specific page types it's not working on. For modern/clean documents it benchmarks very similarly to Google Cloud, but working on supporting older documents better now.

Hello Vik, and thanks for your work on Surya, I really liked it once I found it, but my main issue now is the latency and hardware requirements, as accuracy could be fixed overtime for different page types.

For example, I'm deploying tahweel to one of my webapps to allow limited number of users to run OCR on PDF files. I'm using a small CPU machine for this, deploying Surya will not be the same and I think you are facing similar issues in https://www.datalab.to.

Post reply on HN