Replace OCR with Vision Language Models
github.com
Replace OCR with Vision Language Models
1–10 of 130 posts
Re: Replace OCR with Vision Language Models
#2I wonder what the speed of this approach vs traditional ocr techniques. Also, curious if this could be used for text detection (find a bounding box containing text within an image).
Re: Replace OCR with Vision Language Models
#3Can I use this to convert flowcharts to yaml representations?
Re: Replace OCR with Vision Language Models
#4I think OCR tools are good at what they say on the box, recognizing characters on a piece of paper etc. If I understand this right, the advantage of using a vision language model is the added logic that you can say things like: "Clearly this is a string, but does it look like a timestamp or something else?"
Re: Replace OCR with Vision Language Models
#5I've been experimenting with vlm-run (plus custom form definitions), and it works surprisingly well with Gemini 2.0 Flash. Costs, as I understand, are also quite low for Gemini. You'll have best results with simple to medium-complexity forms, roughly the same ones you could ask a human to process with less than 10 minutes of training.
If you need something like this, it's definitely good enough that you should consider kicking the tires.
Re: Replace OCR with Vision Language Models
#6If you just want to play with using a vision model to do OCR, I made a little script that uses KoboldCpp to do it locally.
Re: Replace OCR with Vision Language Models
#7What's the characters-per-Wh of an LLM compared to traditional OCR?
Re: Replace OCR with Vision Language Models
#8I've been experimenting with vlm-run (plus custom form definitions), and it works surprisingly well with Gemini 2.0 Flash. Costs, as I understand, are also quite low for Gemini. You'll have best results with simple to medium-complexity forms, roughly the same ones you could ask a human to process with less than 10 minutes of training. If you need something like this, it's definitely good enough that you should consid…
Very cool! If you have more examples / schemas you'd be interested in sharing, feel free to add to the `contrib` section.
Re: Replace OCR with Vision Language Models
#9I wonder what the speed of this approach vs traditional ocr techniques. Also, curious if this could be used for text detection (find a bounding box containing text within an image).
Was just coming here to say this, there does not yet exist a multimodal vision LLM approach that is capable of identifying bounding boxes of where the text occurs. I suppose you could manually cut the image up and send each part separately to the LLM but that feels like an kludge and it's still in-exact.
Re: Replace OCR with Vision Language Models
#10Not really interested until this can run locally without api keys :\