Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

github.com

Post reply on HN
Print view View original

111–120 of 178 posts

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#111

Post by ozim »

janalsncm wrote: post #26

Having tried this in the past, it can work pretty well 90% of the time. However, there are still some areas it will struggle. Imagine you are trying to read a lease contract. The two areas which the LLM may be useless are numbers and names (names of people or places/addresses). There’s no way for your LLM to accurately know what the rent should be, or to know about the name of a specific person.

Why does it have to be 100% accurate?

If you get 90% of work done and you have to fix some numbers and names it still saves you time, isn't it?

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#112

Post by themanmaran »

jdthedisciple wrote: post #101

Very recently we had Zerox [0] (Pdf -> Image -> GPT4o-mini based OCR) and I found it to work fantastically well) Would be curious about comparisons between these. [0] https://github.com/getomni-ai/zerox

Hey, author of Zerox here! Happy to hear you saw good results.

Given both libraries are primarily using GPT-4o-mini, I'd expect pretty similar performance. The author here seems to have some better prompts than we do: https://github.com/Dicklesworthstone/llm_aided_ocr/blob/main...

In our approach, we're just zero shot asking for markdown from the image. Vs this approach of passing in the Tesseract result + image context and asking for correction. I'm curious if there is a meaningful accuracy difference.

My first thought it that the tesseract result may decrease accuracy, especially with tables or multi column pdfs. The tesseract model has a tendency to take everything from a table and throw it into one text blob. So while it's added context to the model, it's often not helpful.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#113

Post by jszymborski »

ozim wrote: post #109
jszymborski wrote: post #93

Earlier quoted context omitted.

Maybe a pipeline like: 1. Segment document: Identify which part of the document is text, what is an image, what is a formula, what is a table, etc... 2. For text, do OCR + LLM. You can use LLMs to calculate the expectation of the predicted text, and if it is super off, try using ViT or something to OCR. 3. For tables, you can get a ViT/CNN to identify the cells to recover positional information, and then OCR + LLM fo…

I don't see how you make LLM improve tables where most of the time table is single word or single value that doesn't have continuous context like a sentence.

IMHO, the LLM correction is most relevant/useful in the edge cases rather than the modal ones, so I totally agree.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#114

Post by wahnfrieden »

fred123 wrote: post #106
wahnfrieden wrote: post #99

Earlier quoted context omitted.

Sadly no bounding rects

You can get them through the Vision API (Swift/Objective-C/AppleScript)

Yes but it's relatively shit

The Vision API can't even read vertical Japanese text

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#115

Post by jesprenj »

> My original project had all sorts of complex stuff for detecting hallucinations and incorrect, spurious additions to the text (like "Here is the corrected text" preambles

> asks it to correct OCR errors

So, if I understand correctly, you add some prompt like "fix this text" and then the broken text?

Why don't you do it differently, by not using a chat model but instead a completion model and input the broken OCRd text in the model token by token and then get next token probabilities and then select the token that matches the original document as best as possible, maybe looking 3-5 tokens in advance?

Wouldn't this greatly decrease "hallucinations"?

I'm not trying to insult your approach, I'm just asking for your opinion.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#116

Post by 47282847 »

esafak wrote: post #77
eigenvalue wrote: post #73

Earlier quoted context omitted.

Not sure how you would do that without having the ground truth to compare to. It's also very hard to measure once you start messing with the formatting (like converting it to markdown or suppressing page numbers and repeated headers/footers). I think it would also vary a lot depending on the quality of the original scan and the format and content of the document. There's really no substitute from just trying it on yo…

You can correct the transcript to create the ground truth. Or print your own document, then run OCR on it. OCR evaluation has been a thing for decades. edit: Better than a single document, process a standard OCR dataset: https://paperswithcode.com/task/optical-character-recognitio...

Standard datasets can no longer be used for benchmarking against LLMs since they have already been fed into it and are thus too well-known to compare to lesser known documents.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#117

Post by choilive »

ozim wrote: post #111
janalsncm wrote: post #26

Having tried this in the past, it can work pretty well 90% of the time. However, there are still some areas it will struggle. Imagine you are trying to read a lease contract. The two areas which the LLM may be useless are numbers and names (names of people or places/addresses). There’s no way for your LLM to accurately know what the rent should be, or to know about the name of a specific person.

Why does it have to be 100% accurate? If you get 90% of work done and you have to fix some numbers and names it still saves you time, isn't it?

Theres some time savings, but not a ton.

If theres 30 fields on a document @ 90% accuracy - each field would still need to be validated by a human because you can't trust that it is correct. So the O(n) human step of checking each field is still there, and for fields that are long strings that are pseudo-random looking (think account numbers, numbers on invoices and receipts, instrumentation measurement values, etc.) there is almost no time savings because the mental effort to input something like 015729042 is about the same as verifying it is correct.

At 100% accuracy you remove that need altogether.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#118

Post by aaron695 »

troysk wrote: post #63

In my experience, this works well but doesn't scale to all kinds of documents. For scientific papers; it can't render formulas. meta's nougat is the best model to do that. For invoices and records; donut works better. Both these models will fail in some cases so you end up running LLM to fix the issues. Even with that LLM won't be able to do tables and charts justice, as the details were lost during OCR process (bold…

[dead]

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#119

Post by stavros »

dcastm wrote: post #102
stavros wrote: post #96

Earlier quoted context omitted.

If you're attaching the image, why even send the text? It can read it, no?

To reduce hallucinations. https://arxiv.org/pdf/2311.03287

Huh, very interesting, thank you. In my (limited) tests with GPT-4 vision (before 4o), it did really well with just the image, but it's good to know that the OCR can help.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#120

Post by EarlyOom »

troysk wrote: post #63

In my experience, this works well but doesn't scale to all kinds of documents. For scientific papers; it can't render formulas. meta's nougat is the best model to do that. For invoices and records; donut works better. Both these models will fail in some cases so you end up running LLM to fix the issues. Even with that LLM won't be able to do tables and charts justice, as the details were lost during OCR process (bold…

We've been trying to solve this with https://vlm.run: the idea is to combine the character level accuracy of an OCR pipeline (like Tesseract) with the flexibility of a VLM. OCR pipelines struggle with non-trivial text layouts and don't have any notion of document structure, which means there needs to be another layer on top to actually extract text content to the right place. At the other end of the spectrum, VLMs (like GPT4o) tend to perform poorly on things like dense tables (either hallucinating or giving up entirely) and complex forms, in addition to being much slower/more expensive. Part of the fix is to allow a 'manager' VLM to dispatch to OCR on dense, simple documents, while running charts, graphs etc. through the more expensive VLM pipeline.
Post reply on HN