Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Qwen-2.5-32B is now the best open source OCR model

github.com

Post reply on HN
Print view View original

21–30 of 51 posts

Re: Show HN: Qwen-2.5-32B is now the best open source OCR model

#21

Post by kapitalx »

kapitalx wrote: post #20
ks2048 wrote: post #15

I suppose none of these models can output bounding box coordinates for extracted text? That seems to be a big advantage of traditional OCR over LLMs. For applications I'm interested in, until we can get to 95+% accuracy, it will require human double-checking / corrections, which seems unfeasible w/o bounding boxes to quickly check for errors.

If you're limited to open source models, that's very true. But for larger models and depending on your document needs, we're definitely seeing very high accuracy (95%-99%) for direct to json extraction (no markdown in between step) with our solution at https://doctly.ai .

In addition, gemini Pro 2.5 does really well with bounding boxes, but yeah not open source :(

Re: Show HN: Qwen-2.5-32B is now the best open source OCR model

#22

Post by themanmaran »

dylan604 wrote: post #16
themanmaran wrote: post #10

Earlier quoted context omitted.

Yes, I'll add that to the writeup! You're right, initially excluded it because it was really dependent on the providers, so lots of variance. Especially with the Qwen models. High level results were: - Qwen 32b => $0.33/1000 pages => 53s/page - Qwen 72b => $0.71/1000 pages => 51s/page - Llama 90b => $8.50/1000 pages => 44s/page - Llama 11b => $0.21/1000 pages => 08s/page - Gemma 27b => $0.25/1000 pages => 22s/page -…

One of these things is not like the others. $8.50/1000?? Any chance that's a typo? Otherwise, for someone that has no experience with LLM pricing models, why is Llama 90b so expensive?

That was the cost when we ran Llama 90b using TogetherAI. But it's quite hard to standardize, since it depends a lot on who is hosting the model (i.e. together, openrouter, grok, etc.)

I think in order to run a proper cost comparison, we would need to run each model on an AWS gpu instance and compare the runtime required.

Re: Show HN: Qwen-2.5-32B is now the best open source OCR model

#24

Post by fpgaminer »

I've been consistently surprised by Gemini's OCR capabilities. And yeah, Qwen is climbing the vision ladder _fast_.

In my workflows I often have multiple models competing side-by-side, so I get to compare the same task executed on, say, 4o, Gemini, and Qwen. And I deal with a very wide range of vision related tasks. The newest Qwen models are not only overall better than their previous release by a good margin, but also much more stable (less prone to glitching) and easier to finetune. I'm not at all surprised they're topping the OCR benchmark.

What bugs me though is OpenAI. Outside of OCR, 4o is still king in terms of overall understanding of images. But 4o is now almost a year old, and in all that time they have neither improved the vision performance in any newer releases, nor have they improved OCR. OpenAI's OCR has been bad for a long time, and it's both odd and annoying.

Taken with a grain of salt since again I've only had it in my workflow for about a week or two, but I'd say Qwen 2.5 VL 72b beats Gemini for general vision. That lands it in second place for me. And it can be run _locally_. That's nuts. I'm going to laugh if Qwen drops another iteration in a couple months that beats 4o.

Re: Show HN: Qwen-2.5-32B is now the best open source OCR model

#25

Post by michaelt »

ks2048 wrote: post #15

I suppose none of these models can output bounding box coordinates for extracted text? That seems to be a big advantage of traditional OCR over LLMs. For applications I'm interested in, until we can get to 95+% accuracy, it will require human double-checking / corrections, which seems unfeasible w/o bounding boxes to quickly check for errors.

qwen2.5-vl-72b-instruct seems perfectly happy outputting bounding boxes in my testing.

There's also a paper https://arxiv.org/pdf/2409.12191 where they explicitly say some of their training included bounding boxes and coordinates.

Re: Show HN: Qwen-2.5-32B is now the best open source OCR model

#27

Post by azinman2 »

kapitalx wrote: post #23
azinman2 wrote: post #3

News update: OCR company touts new benchmark that shows its own products are the most performant.

To be fair, they didn't include themselves at all in the graph.

They did. It’s in the #1 spot

Update: looks like the removed themselves from the graph since I saw it earlier today!

Re: Show HN: Qwen-2.5-32B is now the best open source OCR model

#28

Post by int_19h »

dylan604 wrote: post #16
themanmaran wrote: post #10

Earlier quoted context omitted.

Yes, I'll add that to the writeup! You're right, initially excluded it because it was really dependent on the providers, so lots of variance. Especially with the Qwen models. High level results were: - Qwen 32b => $0.33/1000 pages => 53s/page - Qwen 72b => $0.71/1000 pages => 51s/page - Llama 90b => $8.50/1000 pages => 44s/page - Llama 11b => $0.21/1000 pages => 08s/page - Gemma 27b => $0.25/1000 pages => 22s/page -…

One of these things is not like the others. $8.50/1000?? Any chance that's a typo? Otherwise, for someone that has no experience with LLM pricing models, why is Llama 90b so expensive?

It's not uncommon when using brokers to see outliers like this. What happens basically is that some models are very popular and have many different providers, and are priced "close to the metal" since the routing will normally pick the cheapest option with the specified requirements (like context size). But then other models - typically more specialized ones - are only hosted by a single provider, and said provider can then price it much higher than raw compute cost.

E.g. if you look at https://openrouter.ai/models?order=pricing-high-to-low, you'll see that there are some 7B and 8B models that are more expensive than Claude Sonnet 3.7.

Re: Show HN: Qwen-2.5-32B is now the best open source OCR model

#30

Post by stavros »

CSMastermind wrote: post #11

I've been very impressed with Qwen in my testing, I think people are underestimating it

I wrote a small, client-side-JS-only app that does OCR and TTS on board game cards, so my friends and I can listen to someone read the cards' flavor text. On a few pages of text in total so far, Qwen has made zero mistakes. It's very impressive.
Post reply on HN