71–80 of 194 posts
Ironically, the pill counting example selected to showcase "the best vision model" can be easily solved with OpenCV template matching, a technology created 25 years ago.
That is certainly pretty far from what was possible 25 years ago.
When reasoning got introduced a year ago to GPT 5, on average the model performed worse than GPT4-o for short video clip captioning (Ie hallucinating actions that didn’t happen). The old GPT 5 was extremely finicky in terms of fps sample rate.
The other SOTA LLMs (like Gemini Pro) have clearly been optimized for long video understanding, since they can’t see almost anything sub-second (even if you up the frame sampling rate).
Sol is the first model we’ve seen to accurately caption complex sub-second movements (eg woman suddenly turns heard head to right). It’s robust to different fps sample rates so I can only guess that they trained on videos sampled at different fps.
Anecdotal but I've seen it use python to crop, zoom, and "enhance" (fiddle with sharpness and brightness) images to read sections of handwritten census data from the 1800s. Feels like that there might just be a mismatch of capabilities when it comes to straight outputting coordinates but I bet the model is better at actually finding the answer given any tools available. Which I get is a bit of an apples and oranges situation.
I've also tried to use it to identify an old pair of glasses and it didn't stand a chance, so I do think it's not quite there yet when it comes to some vision tasks.
Also it seems to be more capable, need to test more, but I think it's at least getting on par and it's fully open-source and open-weights.
Here's some benchmarks:
https://benchlm.ai/compare/gpt-5-6-sol-vs-qwen3-8-max
https://qwen.ai/blog?id=qwen3.8#full-benchmark-table (incredible UI/UX demos)
https://venturebeat.com/technology/qwen3-8-max-arrives-with-...
EDIT: Am I early to the discussion, or is none else using Qwen3.8-max?
I run the free service https://countrx.app/ so i have some idea what goes into counting. The performance as a general model is indeed really impressive and i think they might actually win compared to fine tuned models. Their feedback loop of training on user data is incredibly strong. I've learned that lots of accuracy results depends on threshold configs, which llms should be able to dynamically set. Or the future w…
Earlier quoted context omitted.
Yeah I was thinking about giving Luna a go with my PDF data extraction, but I think I‘ll stay on Gemini. It does a very good job.
Gemini is still my top choice within production software for typical data extraction from unstructured data. Gemini Flash Lite feels like a cheat code for speed, and it's really cheap. Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.
Ironically, the pill counting example selected to showcase "the best vision model" can be easily solved with OpenCV template matching, a technology created 25 years ago.
I understand why you would like to use an LLM for vision. I do it myself often enough. I don't understand however, why the pill detection and counting is included in this benchmark. That is a task which you would perform with OpenCV right? In my personal mini benchmark minicpm-v-4.6 scores amazingly well. Its a 0.8B model which runs fine on many consumer hardware.
Are any of these vision benchmarks binocular in order to introduce depth perception? I keep waiting for these AI companies to assemble the parts into a great autonomous driving module.
Earlier quoted context omitted.
Yeah I was thinking about giving Luna a go with my PDF data extraction, but I think I‘ll stay on Gemini. It does a very good job.
Gemini is still my top choice within production software for typical data extraction from unstructured data. Gemini Flash Lite feels like a cheat code for speed, and it's really cheap. Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.