131–140 of 178 posts
I dont want hallucinations in places where OCR loses plot. I want either better OCR or error message telling me to repeat the scan.
> My original project had all sorts of complex stuff for detecting hallucinations and incorrect, spurious additions to the text (like "Here is the corrected text" preambles > asks it to correct OCR errors So, if I understand correctly, you add some prompt like "fix this text" and then the broken text? Why don't you do it differently, by not using a chat model but instead a completion model and input the broken OCRd t…
Is there any goodmodel for OCR but on handwritten information? I feel like most models are currently kind of trash
Model: https://huggingface.co/microsoft/Florence-2-large
Demo space: https://huggingface.co/spaces/gokaygokay/Florence-2
Even simpler, you can convert each PDF page to a PNG and ask gpt4 to simply transcribe the image. In my experience it's extremely accurate, more so than Tesseract or classic OCR.
That would cost like 100x as much though.
It's honestly shocking how much gpt4o with vision has simplified things.
Earlier quoted context omitted.
You can get them through the Vision API (Swift/Objective-C/AppleScript)
Yes but it's relatively shit The Vision API can't even read vertical Japanese text
paging Jason Huggins ( https://news.ycombinator.com/user?id=hugs ) to add his two cents to this discussion
I'm extremely excited by real, non-hype reasons to use LLMs, and I've also been frustrated that OCR isn't 100% accurate... I currently use Tesseract OCR in the context of UI automation of mobile apps. UI automation is already notorious for flakiness, I don't need to add to the problem... BUT... sometimes you only have access to the visible screen and literally nothing else... or you're in a regulated environment like payments, automative, or medical device testing where you're required to test the user interface exactly the way a user would, and you still want to automate that -- in those cases, all options are on the table, especially if an LLM-backed-OCR approach works better.
But with all that said, my "acid test" for any multimodal LLM here is to "simply" find the X,Y coordinates of "1", "2", "+", and "=" on the screenshot of a calculator app. So far in my testing, with no or minimal extra prompt engineering... Chat-GPT4o and Llava 1.5 fail this test miserably. But based on the pace of AI announcements these days, I look forward to this being a solved problem in a few months? Or... is the LLM-Aided OCR Project the magic I've been looking for? Tools like plain Tesseract and EasyOCR retain the X,Y locations in the scanned document image of the source text. I can't tell if that meta-information is lost when run through the LLM here.
Earlier quoted context omitted.
Maybe a pipeline like: 1. Segment document: Identify which part of the document is text, what is an image, what is a formula, what is a table, etc... 2. For text, do OCR + LLM. You can use LLMs to calculate the expectation of the predicted text, and if it is super off, try using ViT or something to OCR. 3. For tables, you can get a ViT/CNN to identify the cells to recover positional information, and then OCR + LLM fo…
I don't see how you make LLM improve tables where most of the time table is single word or single value that doesn't have continuous context like a sentence.
In my experience, this works well but doesn't scale to all kinds of documents. For scientific papers; it can't render formulas. meta's nougat is the best model to do that. For invoices and records; donut works better. Both these models will fail in some cases so you end up running LLM to fix the issues. Even with that LLM won't be able to do tables and charts justice, as the details were lost during OCR process (bold…
Maybe a pipeline like: 1. Segment document: Identify which part of the document is text, what is an image, what is a formula, what is a table, etc... 2. For text, do OCR + LLM. You can use LLMs to calculate the expectation of the predicted text, and if it is super off, try using ViT or something to OCR. 3. For tables, you can get a ViT/CNN to identify the cells to recover positional information, and then OCR + LLM fo…
I prefer to do all of this in 1 step with an LLM with a good prompt and few shots.
With so many passes with images, the costs/time will be high with ViT being slower.