I've been very impressed with Qwen in my testing, I think people are underestimating it
Show HN: Qwen-2.5-32B is now the best open source OCR model
11–20 of 51 posts
Re: Show HN: Qwen-2.5-32B is now the best open source OCR model
#12Earlier quoted context omitted.
Someone should try to reproduce and post it here. I can't, my PC is about 15 years old. :( (It is not a joke.)
Reproducing the whole benchmark would be expensive, OmniAi starts at $250/month.
Generally running the whole benchmark is ~$200, since all the providers cost money. But if anyone wants to specifically benchmark Omni just drop us a note and we'll make the credits available.
Re: Show HN: Qwen-2.5-32B is now the best open source OCR model
#13Re: Show HN: Qwen-2.5-32B is now the best open source OCR model
#14I've been doing some experiments with the OCR API on macOS lately and wonder how it compares to these LLMs.
Overall, it's very impressive, but makes some mistakes (on easy images - i.e. obviously wrong) that require human intervention.
I would like to compare it to these models, but this benchmark is beyond OCR - extracted structured JSON.
Re: Show HN: Qwen-2.5-32B is now the best open source OCR model
#15I suppose none of these models can output bounding box coordinates for extracted text? That seems to be a big advantage of traditional OCR over LLMs.
For applications I'm interested in, until we can get to 95+% accuracy, it will require human double-checking / corrections, which seems unfeasible w/o bounding boxes to quickly check for errors.
Re: Show HN: Qwen-2.5-32B is now the best open source OCR model
#16You mention that you measured cost and latency in addition to accuracy - would you be willing to share those results as well? (I understand that for these open models they would vary between providers, but it would be useful to have an approximate baseline.)
Yes, I'll add that to the writeup! You're right, initially excluded it because it was really dependent on the providers, so lots of variance. Especially with the Qwen models. High level results were: - Qwen 32b => $0.33/1000 pages => 53s/page - Qwen 72b => $0.71/1000 pages => 51s/page - Llama 90b => $8.50/1000 pages => 44s/page - Llama 11b => $0.21/1000 pages => 08s/page - Gemma 27b => $0.25/1000 pages => 22s/page -…
One of these things is not like the others. $8.50/1000?? Any chance that's a typo? Otherwise, for someone that has no experience with LLM pricing models, why is Llama 90b so expensive?
Re: Show HN: Qwen-2.5-32B is now the best open source OCR model
#17Re: Show HN: Qwen-2.5-32B is now the best open source OCR model
#18You mention that you measured cost and latency in addition to accuracy - would you be willing to share those results as well? (I understand that for these open models they would vary between providers, but it would be useful to have an approximate baseline.)
Yes, I'll add that to the writeup! You're right, initially excluded it because it was really dependent on the providers, so lots of variance. Especially with the Qwen models. High level results were: - Qwen 32b => $0.33/1000 pages => 53s/page - Qwen 72b => $0.71/1000 pages => 51s/page - Llama 90b => $8.50/1000 pages => 44s/page - Llama 11b => $0.21/1000 pages => 08s/page - Gemma 27b => $0.25/1000 pages => 22s/page -…
A 2d plot would be great
Re: Show HN: Qwen-2.5-32B is now the best open source OCR model
#19I suppose none of these models can output bounding box coordinates for extracted text? That seems to be a big advantage of traditional OCR over LLMs. For applications I'm interested in, until we can get to 95+% accuracy, it will require human double-checking / corrections, which seems unfeasible w/o bounding boxes to quickly check for errors.
I'd guess that it wouldn't be a huge effort to fine tune them to produce bounding boxes.
I haven't done it with OCR tasks, but I have fine tuned other models to produce them instead of merely producing descriptive text. I'm not sure if there are datasets for this already, but creating one shouldn't be very difficult.
Re: Show HN: Qwen-2.5-32B is now the best open source OCR model
#20I suppose none of these models can output bounding box coordinates for extracted text? That seems to be a big advantage of traditional OCR over LLMs. For applications I'm interested in, until we can get to 95+% accuracy, it will require human double-checking / corrections, which seems unfeasible w/o bounding boxes to quickly check for errors.
If you're limited to open source models, that's very true. But for larger models and depending on your document needs, we're definitely seeing very high accuracy (95%-99%) for direct to json extraction (no markdown in between step) with our solution at https://doctly.ai.