Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace OCR with Vision Language Models

github.com

Post reply on HN
Print view View original

71–80 of 130 posts

Re: Replace OCR with Vision Language Models

#71

Post by rafram »

hashta wrote: post #47
EarlyOom wrote: post #34

Earlier quoted context omitted.

This is the main focus of VLM Run and typed extraction more generally. If you provide proper type constraints (e.g. with Pydantic) you can dramatically reduce the surface area for hallucination. Then there's actually fine-tuning on your dataset (we're working on this) to push accuracy beyond what you get from an unspecialized frontier model.

An effective way that usually increases accuracy is to use an ensemble of capable models that are trained independently (e.g., gemini, gpt-4o, qwen). If >x% of them have the same output, accept it, otherwise reject and manually review

There’s a very low chance that three separate models will come up with the same result. There are always going to be errors, small or large. Even if you find a way around that, running the process three times on every page is going to be prohibitively expensive, especially if you want to finetune.

Re: Replace OCR with Vision Language Models

#72

Post by K0balt »

erulabs wrote: post #46

You sort of have to use both. OCR and LLM and then correlate the two results. They are bad at very different things, but a subsequent call to a 2nd LLM to pair together the results does improve quality significantly, plus you get both document understanding and context as well as bounding boxes, etc. I'm building a "never fill out paperwork again" app, if anyone is interested, would be happy to chat!

A VLM that invokes ocr tool use is a compelling idea that could result in pretty good results, I would expect.

Re: Replace OCR with Vision Language Models

#74

Post by fzysingularity »

rafram wrote: post #69
EarlyOom wrote: post #34

Earlier quoted context omitted.

This is the main focus of VLM Run and typed extraction more generally. If you provide proper type constraints (e.g. with Pydantic) you can dramatically reduce the surface area for hallucination. Then there's actually fine-tuning on your dataset (we're working on this) to push accuracy beyond what you get from an unspecialized frontier model.

Re type constraints: Not really. If one of the fields in my JSON is `name` but the model can’t read the name on the page, it will very happily make one up. Type constraints are good for making sure that your data is parseable, but they don’t do anything to fix the undetectable inaccuracy problem. Fine-tuning does help, though.

Yes, both false positives and false negatives like the one you mentioned happens when the schema is sometimes ill-defined. Making name optional via `name: str | None` actually turns out ensure that the model only fills it if it’s certain that field exists.

These are some of the nuances we had to work with during VLM fine-tuning with structured JSON.

Re: Replace OCR with Vision Language Models

#75

Post by staticman2 »

rafram wrote: post #32

It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…

I think it would be pretty reliable in controlled circumstances. If I take a picture of a book with my cell phone- google Gemini pro is much better at recognizing the text than Samsung's built in OCR.

Re: Replace OCR with Vision Language Models

#76

Post by Grimblewald »

staticman2 wrote: post #75
rafram wrote: post #32

It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…

I think it would be pretty reliable in controlled circumstances. If I take a picture of a book with my cell phone- google Gemini pro is much better at recognizing the text than Samsung's built in OCR.

I would think the same, the cause for hesitation is that we only think this, but cannot know it without thorough testing. Right now the scope of problems where things behave reliably and as expected and scope of problems where things get whacky are unknown. The borders are known to some rather fuzzy extent at best, by people who work with these things as a full-time job. This means we are just blindly gambling on it. For important things, archiving, etc. where truth matters, I will continue using traditional OCR until we can define the reliable use-case scope of LLM based OCR better. I am extremely enthusiastic about LLM's and the things these offer, but i am also a realist. LLM's are an infant technology, and no-where near the level of maturity that companies like openAI claim.

Re: Replace OCR with Vision Language Models

#78

Post by wahnfrieden »

acdha wrote: post #57
iLemming wrote: post #56

What's the fastest and accurate CLI OCR tool? My use case is simple - I want to be able to grab a piece of screen (Flameshot is great for that), and OCR it. I need this for note-taking during pair-programming over Zoom. Currently I'm using tesseract - it works, it's fast, but it also makes mistakes; it would be also great if it could discern tabular data and put them in ascii or markdown tables. I've tried docling, b…

Anything using the Apple Vision framework is fast and surprisingly accurate: https://github.com/bytefer/macos-vision-ocr

This uses the old APIs that are less accurate than the new Swift-only LiveText ones

Re: Replace OCR with Vision Language Models

#79

Post by Inviz »

Service doesnt inspire confidence. Openai-compatible api doesnt work (expects `content.str` in message to be a string - ???). Getting 500s on non-openai compatible endpoint - seems like timeouts(?). When it did work it missed a lot, and hallucinated a lot too on custom documents/schemas.

Re: Replace OCR with Vision Language Models

#80

Post by constantinum »

Hackbraten wrote: post #63
constantinum wrote: post #59

Earlier quoted context omitted.

The primary issue with LLMs is hallucination, which can lead to incorrect data and flawed business decisions. For example, Llamaparse( https://docs.llamaindex.ai/en/stable/llama_cloud/llama_parse... ) uses LLMs for PDF text extraction but faces hallucination problems. See this issue for more details: https://github.com/run-llama/llama_parse/issues/420 . For those interested, try LLMWhisperer( https://unstract.com/llm…

> try LLMWhisperer( https://unstract.com/llmwhisperer/ ) for OCR. It avoids LLMs The website you linked says it uses LLMs?

The tool doesn't use any LLMs for processing/parsing the data. It parses and converts into raw text.

The final output(raw text) of the parsing is then fed to LLMs for data extraction. e.g. Extracting data from insurance, banking, and invoice documents.

Post reply on HN