Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

github.com

Post reply on HN
Print view View original

141–150 of 178 posts

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#141

Post by jdthedisciple »

themanmaran wrote: post #112
jdthedisciple wrote: post #101

Very recently we had Zerox [0] (Pdf -> Image -> GPT4o-mini based OCR) and I found it to work fantastically well) Would be curious about comparisons between these. [0] https://github.com/getomni-ai/zerox

Hey, author of Zerox here! Happy to hear you saw good results. Given both libraries are primarily using GPT-4o-mini, I'd expect pretty similar performance. The author here seems to have some better prompts than we do: https://github.com/Dicklesworthstone/llm_aided_ocr/blob/main... In our approach, we're just zero shot asking for markdown from the image. Vs this approach of passing in the Tesseract result + image cont…

Hey there!

Intuitively I would be surprised if the tesseract-way was better to be honest.

I'm not the most experienced with OCR but I have tried Tesseract in the past and had difficulties. Like others mentioned, it would mix up things like 4 vs A etc. and had massive issues with decimal numbers.

I have had zero such issues with zerox.

Someone on HN also recently mentioned that tesseract is OCR of the past and basically outlived by a long shot already, I feel like that might be true.

Btw I feel extremely lucky to have found Zerox because we needed a solution like that at work and so I introduced it and spun up a prototype to solve our problem and it works so well (combined with the brand new "strict json" feature by openai), so really great job making it!

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#142

Post by jszymborski »

troysk wrote: post #140
jszymborski wrote: post #93

Earlier quoted context omitted.

Maybe a pipeline like: 1. Segment document: Identify which part of the document is text, what is an image, what is a formula, what is a table, etc... 2. For text, do OCR + LLM. You can use LLMs to calculate the expectation of the predicted text, and if it is super off, try using ViT or something to OCR. 3. For tables, you can get a ViT/CNN to identify the cells to recover positional information, and then OCR + LLM fo…

How to segment the document without LLM? I prefer to do all of this in 1 step with an LLM with a good prompt and few shots. With so many passes with images, the costs/time will be high with ViT being slower.

Segmenting can likely be done on a really small resolution and with a CNN, making it real short.

There are some heuristic ways of doing it but i doubt you'll be able to distinguish equations from text.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#143

Post by arresin »

hugs wrote: post #138
localfirst wrote: post #104

paging Jason Huggins ( https://news.ycombinator.com/user?id=hugs ) to add his two cents to this discussion

Oh hey! (This might be the first time I've been paged on HN) I'm extremely excited by real, non-hype reasons to use LLMs, and I've also been frustrated that OCR isn't 100% accurate... I currently use Tesseract OCR in the context of UI automation of mobile apps. UI automation is already notorious for flakiness, I don't need to add to the problem... BUT... sometimes you only have access to the visible screen and litera…

> But with all that said, my "acid test" for any multimodal LLM here is to "simply" find the X,Y coordinates of "1", "2", "+", and "=" on the screenshot of a calculator app.

hugs if you find such a thing, could you please make a post about it? I am looking for the same thing and try the same test.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#144

Post by constantinum »

Most of the document processing automation projects at an enterprise level require parsing of complex documents with tables, forms, handwriting, checkboxes, scanned documents. Example includes ACORD insurance forms, IRS tax forms and bank statements. I’m not even getting into how different each document can be even if they are of the same nature.

For any one curious on automating document processing end-to-end by leveraging llms do try Unstract. It is opens source.

https://github.com/Zipstack/unstract

Unstract also has a commercial version of document agnostic parser which you can channel to any RAG projects.

https://unstract.com/llmwhisperer/

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#147

Post by mewpmewp2 »

collinmcnulty wrote: post #146

Does anyone have a solution that works well for handwriting? I have 10 years of handwritten notes that I’d love to make searchable but all OCR I’ve tried has been quite poor. These solutions seem focused on typeset documents.

Have you tried GPT 4o. Some handwriting it seems to do very well.

Just test out by pasting the image in ChatGPT interface and asking what is written there.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#148

Post by mewpmewp2 »

Sidneys1 wrote: post #100
katzinsky wrote: post #59

Vision transformers are good enough that you can use them alone even on cursive handwriting. I've had amazing results with Microsoft's models and have my own little piece of wrapper software I use to transcribe blog posts I write in my notebook.

I'd like to hear more about this! I keep coming back to trying to OCR my journals, but nothing I've tried so far works well (enough) on handwriting.

Have you tried uploading image of your handwriting to ChatGPT interface with ChatGPT 4o?

And what the results were? And if not could you try and let us know what the results are.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#150

Post by hugs »

arresin wrote: post #143
hugs wrote: post #138

Earlier quoted context omitted.

Oh hey! (This might be the first time I've been paged on HN) I'm extremely excited by real, non-hype reasons to use LLMs, and I've also been frustrated that OCR isn't 100% accurate... I currently use Tesseract OCR in the context of UI automation of mobile apps. UI automation is already notorious for flakiness, I don't need to add to the problem... BUT... sometimes you only have access to the visible screen and litera…

> But with all that said, my "acid test" for any multimodal LLM here is to "simply" find the X,Y coordinates of "1", "2", "+", and "=" on the screenshot of a calculator app. hugs if you find such a thing, could you please make a post about it? I am looking for the same thing and try the same test.

yes!
Post reply on HN