Was it trained by distilling some other model?
GLM-OCR – A multimodal OCR model for complex document understanding
71–80 of 82 posts
Re: GLM-OCR – A multimodal OCR model for complex document understanding
#72I've been trying different OCR models on what should be very simple - subtitles (these are simple machine-rendered text). While all models do very well (95+% accuracy), I haven't seen a model not occasionally make very obvious mistakes. Maybe it will take a different approach to get the last 1%...
"95+% accuracy"
That doesn't sound great
Re: GLM-OCR – A multimodal OCR model for complex document understanding
#73Earlier quoted context omitted.
Interesting. What kind of layout do you have? My documents have one or two-column layouts, often inconsistently across pages or even within a page (which tripped older layout detection methods). Most models seem to understand that well enough so they are good enough for my use case.
Documents that come from FOIA. So, some scanned, some not. Lots of forms and lots of hand writing to add info that the form format doesn't recognize. Lots of repeated documents, but lots of one-off documents that have high signal.
I'd be very curious what works well with FOIA historical documents that have been scanned by hand with redactions by markers & etc.
Re: GLM-OCR – A multimodal OCR model for complex document understanding
#74Earlier quoted context omitted.
Are there leaderboards that you follow or trust? Also, do you have preferred OCR models in your experience? I've had some success with dots.OCR, but I'm only beginning to need to work with OCR.
> Are there leaderboards that you follow or trust? Not for OCR. Regardless of how much some people complain about them, I really do appreciate the effort Artificial Analysis puts into consistently running standardized benchmarks for LLMs, rather than just aggregating unverified claims from the AI labs. I don't think LMArena is that amazing at this point in time, but at least they provide error bars on the ELO and giv…
Here's one from last year. Helpful, but I guess they gave up on updating it.
Re: GLM-OCR – A multimodal OCR model for complex document understanding
#75There are a bunch of new OCR models. I’ve also heard very good things about these two in particular: - LightOnOCR-2-1B: https://huggingface.co/lightonai/LightOnOCR-2-1B - PaddleOCR-VL-1.5: https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5 The OCR leaderboards I’ve seen leave a lot to be desired. With the rapid release of so many of these models, I wish there were a better way to know which ones are actually the be…
The best leader board I have used is ocrarena.ai. I agree it is not detailed enough. I wish people could rate what part of the ocr went well or bad (layout, text recognition, etc). However, my more specific results using custom prompts and my own images on their playground page are relatively closely aligned with the rankings as others have voted.
What more are you looking for?
Re: GLM-OCR – A multimodal OCR model for complex document understanding
#76I tested this pretty extensively and it has a common failure mode that prevents me from using: extracting footnotes and similar from the full text of academic works. For some reason, many of these models are trained in a way that results in these being excluded, despite these document sections often containing import details and context. Both versions of DeepseekOCR have the same problem. Of the others I’ve tested, d…
I can get multiple sets of footnotes (critical + content notes) reliably recognized and categorized using gemini-3-flash-preview. I took 15-20 hours to iterate on my prompt for a specific format. Otherwise it would not produce good enough results. It was a slow process because results from batch did not mirror what I was getting from the chat mode, and you have to wait for batch results while analyzing the last set. There was also a bit of debugging of the batch protocol going on at the same time. Flash is also surprisingly affordable for the results I am getting, 4-5x less than I had anticipated. I gave up on gemini-3-pro pretty quickly because it overthinks and messes things up.
Re: GLM-OCR – A multimodal OCR model for complex document understanding
#77Re: GLM-OCR – A multimodal OCR model for complex document understanding
#78I've been trying different OCR models on what should be very simple - subtitles (these are simple machine-rendered text). While all models do very well (95+% accuracy), I haven't seen a model not occasionally make very obvious mistakes. Maybe it will take a different approach to get the last 1%...
"95+% accuracy" That doesn't sound great
I don't have the numbers right here, but roughly 95% subtitles correct and 99% characters correct (but roughly all of those errors are obvious to human labeler).
Re: GLM-OCR – A multimodal OCR model for complex document understanding
#79Is it possible for such a small model to outperform gemini 3 or is this a case of benchmarks not showing the reality? I would love to be hopeful, but so far an open source model was never better than a closed one even when benchmarks were showing that.
Off the top of my head: for a lot of OCR tasks, it’s kind of worse for the model to be smart. I don’t want my OCR to make stuff up or answer questions — I want to to recognize what is actually on the page.
Yes, but that's context specific. If your goal with OCR to make text indexable and searchable with regular text search, then transcribing "lesser" as "lesfer" is bad. And handwriting can often be so bad that you need context to make the call about what the scribbles actually are trying to say.
Evaluation methods, too, are bad because they don't think critically about what the downstream task is. Word Error Rate and Character Error Rate are terrible metrics for most historical HTR, yet they're what people use because of habit.
It's a bit like how for a long time BLEU was the metric for translation quality. BLEU is based on N-gram similarity to a reference translation, so naturally translation methods based on and targeting N-gram similarity (e.g. pre NN Google translate) did well, and looked much better than they actually were.
Re: GLM-OCR – A multimodal OCR model for complex document understanding
#80There was so many OCR models released in the past few months, all VLM models and yet none of them handle Korean well. Every time I try with a random screenshot (not a A4 document) they just fail at a "simple" task. And funnily enough Qwen3 8B VL is the best model that usually get it right (although I couldn't get the bbox quite well). Even more funny, whatever is running on an iphone locally on cpu is insanely good,…
Gemini crushes almost any major script including CJK, even Flash. Not self-hostable though.