Hmm I know someone adding a nn based ocr to number plate recognition. In production. Why bring llms into this? Because all you have is a hammer?
Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
91–100 of 178 posts
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#92Earlier quoted context omitted.
I ended up using EasyOCR. I assume it is too slow in CPU-only mode.
> I assume it is too slow in CPU-only mode. So you don't have to assume: I gave up after running on 8 cores (Ryzen 7 2700) for 10 days for a single page.
I use a container on a machine with an old quad core i7 and no GPU compute. This should take at most tens of seconds per page.
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#93In my experience, this works well but doesn't scale to all kinds of documents. For scientific papers; it can't render formulas. meta's nougat is the best model to do that. For invoices and records; donut works better. Both these models will fail in some cases so you end up running LLM to fix the issues. Even with that LLM won't be able to do tables and charts justice, as the details were lost during OCR process (bold…
Maybe a pipeline like:
1. Segment document: Identify which part of the document is text, what is an image, what is a formula, what is a table, etc...
2. For text, do OCR + LLM. You can use LLMs to calculate the expectation of the predicted text, and if it is super off, try using ViT or something to OCR.
3. For tables, you can get a ViT/CNN to identify the cells to recover positional information, and then OCR + LLM for recovering the contents of cells
4. For formulas (and formulas in tables), just use a ViT/CNN.
5. For images, you can get a captioning ViT/CNN to caption the photo, if that's desired.
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#94I wonder if you could feed back the results from an LLM into the OCR model to get it to make better decisions. E.g., if it's distinguishing a 1 from an I, the LLM could provide a probability distribution.
Or the other direction. Tesseract can give you confidence levels for the guesses it makes about a symbol (see https://tesseract-ocr.github.io/tessdoc/APIExample.html ). You can give the LLM this information and have it tell you which is right. Here's a demo. === input to ChatGPT 4o === I'm trying to read some text but one of the characters is unclear. I will use the notation [xy] to indicate that I think the characte…
Ah, interesting. I guess you could probably get bounding boxes etc.,. too, so you could pass the image as well to a multimodal model (although maybe it'll just run tesseract behind the scenes lol)
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#95I wonder if you could feed back the results from an LLM into the OCR model to get it to make better decisions. E.g., if it's distinguishing a 1 from an I, the LLM could provide a probability distribution.
Or the other direction. Tesseract can give you confidence levels for the guesses it makes about a symbol (see https://tesseract-ocr.github.io/tessdoc/APIExample.html ). You can give the LLM this information and have it tell you which is right. Here's a demo. === input to ChatGPT 4o === I'm trying to read some text but one of the characters is unclear. I will use the notation [xy] to indicate that I think the characte…
That's a great idea, I should add that to my project. Will investigate.
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#96when I was working with Tesseract, a particular issue I had was its tendency to parse a leading "+" as "4" about half the time. e.g. "+40% ROI" would get parsed as "440% ROI". the font was perfectly fine, the screenshots were crispy PNGs. A LLM can't really correct that. I appreciate that Tesseract exists, and it's mostly fine for non-serious things, but I wouldn't let it anywhere near critical data.
ME: Fix this text extracted with OCR: Return on Investment (RO1) is a crucial metric used to evaluate the efficiency and profitability of an investment. If you have achieved an ROI of 440%, it indicates a strong performance and successful investment strategy. To calculate ROI, you subtract the initial cost of the investment from the final value of the investment, then divide that difference by the initial cost, and m…
If you're attaching the image, why even send the text? It can read it, no?
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#97Earlier quoted context omitted.
You can correct the transcript to create the ground truth. Or print your own document, then run OCR on it. OCR evaluation has been a thing for decades. edit: Better than a single document, process a standard OCR dataset: https://paperswithcode.com/task/optical-character-recognitio...
Oh you meant for just a single benchmarked document. I thought you meant to report that for every document you process. I wouldn't want to mislead people by giving stats on a particular kind of scan/document, because it likely wouldn't carry over in general.
[deleted]
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#98In my experience, this works well but doesn't scale to all kinds of documents. For scientific papers; it can't render formulas. meta's nougat is the best model to do that. For invoices and records; donut works better. Both these models will fail in some cases so you end up running LLM to fix the issues. Even with that LLM won't be able to do tables and charts justice, as the details were lost during OCR process (bold…
I agree that vision models that actually have access to the image are a more sound approach than using OCR and trying to fix it up. It may be more expensive though, and depending on what you're trying to do it may be good enough. What I want to do is reading handwritten documents from the 18th century, and I feel like the multistep approach hits a hard ceiling there. Transkribus is multistep, but the line detecion mo…
I also think it’s probably more effective. Every time hand-crafted tools are better than AI but then the model becomes bigger and AI wins. Think hand crafted image classification to full model or hand crafted language translation to full model.
In this case, the model can already do the OCR and becomes an order of magnitude cheaper per year.
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#99Earlier quoted context omitted.
You know, I’ve really looked hard at what’s out there and haven’t been able to find anything else that’s totally free/open, that runs well on CPU, and which has better quality output than Tesseract. I found a couple Chinese projects but had trouble getting them to work and the documentation wasn’t great. If you have any leads on others to try I’d love to hear about them. One of the benefits of this project is that it…
macOS Live Text is incredible. Mac only though
Sadly no bounding rects
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#100Vision transformers are good enough that you can use them alone even on cursive handwriting. I've had amazing results with Microsoft's models and have my own little piece of wrapper software I use to transcribe blog posts I write in my notebook.
I'd like to hear more about this! I keep coming back to trying to OCR my journals, but nothing I've tried so far works well (enough) on handwriting.