Very recently we had Zerox [0] (Pdf -> Image -> GPT4o-mini based OCR) and I found it to work fantastically well)
Would be curious about comparisons between these.
Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
101–110 of 178 posts
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#102Earlier quoted context omitted.
ME: Fix this text extracted with OCR: Return on Investment (RO1) is a crucial metric used to evaluate the efficiency and profitability of an investment. If you have achieved an ROI of 440%, it indicates a strong performance and successful investment strategy. To calculate ROI, you subtract the initial cost of the investment from the final value of the investment, then divide that difference by the initial cost, and m…
If you're attaching the image, why even send the text? It can read it, no?
To reduce hallucinations.
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#103"real improvements came from adjusting the prompts to make things clearer for the model, and not asking the model to do too much in a single pass" This is spot on, and it's the same as how humans behave. If you give a human too many instructions at once, they won't follow all of them accurately. I spend a lot of time thinking about LLMs + documents, and in my opinion, as the models get better, OCR is soon going to be…
Extend looks great - and your real estate play is very interesting. I’ve been playing around extracting key terms from residential leasehold (condominium-type) agreements. Interested to know if you’re doing this sort of thing?
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#104paging Jason Huggins (https://news.ycombinator.com/user?id=hugs) to add his two cents to this discussion
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#105Unfortunately LLM thrown at OCR doesn't work well with large enough to be useful from what I've been told.
Nothing I've seen here offers anything new to what was attempted
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#106Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#107Vision transformers are good enough that you can use them alone even on cursive handwriting. I've had amazing results with Microsoft's models and have my own little piece of wrapper software I use to transcribe blog posts I write in my notebook.
I'd like to hear more about this! I keep coming back to trying to OCR my journals, but nothing I've tried so far works well (enough) on handwriting.
A couple of other people in the thread are using it too apparently. They're the Microsoft TROCR models. You do need a moderate amount of software to deskew, process, and segment the image before handing it to the model but after that it's typically extremely accurate in my experience.
Setting up my software online and monetizing it is next in the queue after my current side project. Although I haven't checked the model licenses.
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#108when I was working with Tesseract, a particular issue I had was its tendency to parse a leading "+" as "4" about half the time. e.g. "+40% ROI" would get parsed as "440% ROI". the font was perfectly fine, the screenshots were crispy PNGs. A LLM can't really correct that. I appreciate that Tesseract exists, and it's mostly fine for non-serious things, but I wouldn't let it anywhere near critical data.
Yes, I also found that Tesseract has still some serious and obvious problems, and they are probably better fixed at that level instead of in another layer.
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#109In my experience, this works well but doesn't scale to all kinds of documents. For scientific papers; it can't render formulas. meta's nougat is the best model to do that. For invoices and records; donut works better. Both these models will fail in some cases so you end up running LLM to fix the issues. Even with that LLM won't be able to do tables and charts justice, as the details were lost during OCR process (bold…
Maybe a pipeline like: 1. Segment document: Identify which part of the document is text, what is an image, what is a formula, what is a table, etc... 2. For text, do OCR + LLM. You can use LLMs to calculate the expectation of the predicted text, and if it is super off, try using ViT or something to OCR. 3. For tables, you can get a ViT/CNN to identify the cells to recover positional information, and then OCR + LLM fo…
I don't see how you make LLM improve tables where most of the time table is single word or single value that doesn't have continuous context like a sentence.
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#110This is a wonderful idea, but while I appreciate the venerable Tesseract I also think it's time to move on.
I personally use PaddlePaddle and have way better results to correct with LLMs.
With PPOCRv3 I wrote a custom Python implementation to cut books at word-level by playing with whitespace thresholds. It works great for the kind of typesetting found generally on books, with predictable whitespace threshold between words. This is all needed because PPOCRv3 is restricted to 320 x 240 pixels if I recall correctly and produces garbage if you downsample a big image and make a pass.
Later on I converted the Python code for working with the Rockchip RK3399Pro NPU, that is, to C. It works wonderfully. I used PaddleOCR2Pytorch to convert the models to rknn-api first and wrote the C implementation that cuts words on top of the rknn-api.
But with PPOCRv4 I think this isn't even needed, it's a newer architecture and I don't think it is bounded by pixel size restriction. That is, it will work "out of the box" so to speak. With the caveat that PPOCRv3 detection always worked better for me, PPOCRv4 detection model gave me big headaches.