Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Qwen-2.5-32B is now the best open source OCR model

github.com

Post reply on HN
Print view View original

1–10 of 51 posts

Show HN: Qwen-2.5-32B is now the best open source OCR model

#1

Post by themanmaran »

Last week was big for open source LLMs. We got:

- Qwen 2.5 VL (72b and 32b)

- Gemma-3 (27b)

- DeepSeek-v3-0324

And a couple weeks ago we got the new mistral-ocr model. We updated our OCR benchmark to include the new models.

We evaluated 1,000 documents for JSON extraction accuracy. Major takeaways:

- Qwen 2.5 VL (72b and 32b) are by far the most impressive. Both landed right around 75% accuracy (equivalent to GPT-4o’s performance). Qwen 72b was only 0.4% above 32b. Within the margin of error.

- Both Qwen models passed mistral-ocr (72.2%), which is specifically trained for OCR.

- Gemma-3 (27B) only scored 42.9%. Particularly surprising given that it's architecture is based on Gemini 2.0 which still tops the accuracy chart.

The data set and benchmark runner is fully open source. You can check out the code and reproduction steps here:

- https://getomni.ai/blog/benchmarking-open-source-models-for-...

- https://github.com/getomni-ai/benchmark

- https://huggingface.co/datasets/getomni-ai/ocr-benchmark

Show HN: Qwen-2.5-32B is now the best open source OCR model
github.com

Re: Show HN: Qwen-2.5-32B is now the best open source OCR model

#4

Post by jauntywundrkind »

azinman2 wrote: post #3

News update: OCR company touts new benchmark that shows its own products are the most performant.

I searched for any link between OmniAI and Alibaba's Qwen, but I can't find any link. Do you know anything I don't know?

All of these models are open source (I think?). They could presumably build their work on any of these options. It behooves them to pick well. And establish some authority along the way.

Re: Show HN: Qwen-2.5-32B is now the best open source OCR model

#5

Post by daemonologist »

You mention that you measured cost and latency in addition to accuracy - would you be willing to share those results as well? (I understand that for these open models they would vary between providers, but it would be useful to have an approximate baseline.)

Re: Show HN: Qwen-2.5-32B is now the best open source OCR model

#6

Post by jauntywundrkind »

The 32b sounds like it has some useful small tweakers. Tweaks to make output more human friendly, better mathematical reasoning, better fine-grained understanding. https://qwenlm.github.io/blog/qwen2.5-vl-32b/ https://news.ycombinator.com/item?id=43464068

Qwen2.5-VL-72b was released two months ago (to little fanfare in submissions, i think, but some very enthusiastic comments such as rabid enthusiasm for handwriting recognition) already very interesting. Its actually one of the releases that kind of turned me on to AI, that broke through some of my skepticism & grumpiness. There's pretty good release notes detailing capabilities here; well done blog post. https://qwenlm.github.io/blog/qwen2.5-vl/

One thing that really piqued my interest was Qwen HTML output, where it can provide bounding boxes in HTML format for its output. That really closes the loop interestingly to me, makes the output something I can imagine quickly building useful visual feedback around, or using the structured data from easily. I can't imagine an easier to use output format.

Re: Show HN: Qwen-2.5-32B is now the best open source OCR model

#8

Post by rustc »

jauntywundrkind wrote: post #4
azinman2 wrote: post #3

News update: OCR company touts new benchmark that shows its own products are the most performant.

I searched for any link between OmniAI and Alibaba's Qwen, but I can't find any link. Do you know anything I don't know? All of these models are open source (I think?). They could presumably build their work on any of these options. It behooves them to pick well. And establish some authority along the way.

The model with the best accuracy in the linked benchmark is "OmniAI" (OP's company) which looks like a paid model, not open source [1].

[1]: https://getomni.ai/pricing

Re: Show HN: Qwen-2.5-32B is now the best open source OCR model

#9

Post by rustc »

johnisgood wrote: post #7
azinman2 wrote: post #3

News update: OCR company touts new benchmark that shows its own products are the most performant.

Someone should try to reproduce and post it here. I can't, my PC is about 15 years old. :( (It is not a joke.)

Reproducing the whole benchmark would be expensive, OmniAi starts at $250/month.

Re: Show HN: Qwen-2.5-32B is now the best open source OCR model

#10

Post by themanmaran »

daemonologist wrote: post #5

You mention that you measured cost and latency in addition to accuracy - would you be willing to share those results as well? (I understand that for these open models they would vary between providers, but it would be useful to have an approximate baseline.)

Yes, I'll add that to the writeup! You're right, initially excluded it because it was really dependent on the providers, so lots of variance. Especially with the Qwen models.

High level results were:

- Qwen 32b => $0.33/1000 pages => 53s/page

- Qwen 72b => $0.71/1000 pages => 51s/page

- Llama 90b => $8.50/1000 pages => 44s/page

- Llama 11b => $0.21/1000 pages => 08s/page

- Gemma 27b => $0.25/1000 pages => 22s/page

- Mistral => $1.00/1000 pages => 03s/page

Post reply on HN