[deleted]
GPT 5.6 Sol is the best "vision" model OpenAI ever released
121–130 of 194 posts
Re: GPT 5.6 Sol is the best "vision" model OpenAI ever released
#122I run the free service https://countrx.app/ so i have some idea what goes into counting. The performance as a general model is indeed really impressive and i think they might actually win compared to fine tuned models. Their feedback loop of training on user data is incredibly strong. I've learned that lots of accuracy results depends on threshold configs, which llms should be able to dynamically set. Or the future w…
How are you running it for free? Are you self funding or do you have sponsors?
Self funded. It's a custom trained efficient model on CPU so it's borderline free
Re: GPT 5.6 Sol is the best "vision" model OpenAI ever released
#123Anecdotal, opinion: Gpt is really good in vision stuff, or at least their MoE seems to be really cohesive. From my experience Claude models can be really good at language but the moment they need to look at a picture and decide why the design is not good what parts need improvement it degrades a lot. My easiest benchmark is giving them a screenshot of a feature in my app and tell it "identify non-normative UI blocks…
Assessing the subjective quality of a thing is in my experience one of the worst ways to use any LLM.
There's a lot of objective principles and decisions that go into subjective quality; if you don't know the field well, asking LLM for assessment is a good way to discover all that.
Re: GPT 5.6 Sol is the best "vision" model OpenAI ever released
#124Earlier quoted context omitted.
Hi, I’m the author of this blog post. I wrote it about 4 weeks ago, and the VLM world is moving so fast that it’s already kinda outdated. I think Gemini 3.7 Flash might be a better choice now, especially when you factor in the price. Here’s a comparison of the best low-cost models I put together last week. What’s crazy is that Gemini 3.7 Flash is now 50% off on OpenRouter, and this chart doesn’t even account for that…
Curious why you didn't try Gemini 3 pro? That is the model I've been using for OCR entry of handwritten datasheets (JPGS of datasheets, structured JSON output). At my scale, the cost of 3 pro is basically not an issue, but if there are improvements in quality, I'd definitely be willing to explore other models
3 Pro is quickly approaching one year old. There's almost no reason to benchmark it, especially since a new version of Gemini Pro was supposed to be released mid 2026 and hasn't seen the light of day.
Re: GPT 5.6 Sol is the best "vision" model OpenAI ever released
#125Gemini 3 Flash should really be included in this comparison. Or at least 3.7. In most of my testing, 3.5 and 3.6 were both a downgrade in terms of vision capabilities, relative to 3, and at a much higher cost. 3.7 is slightly better than 3, finally.
3 Flash never left "preview" status and is listed as deprecated.
Re: GPT 5.6 Sol is the best "vision" model OpenAI ever released
#126Earlier quoted context omitted.
Yeah I was thinking about giving Luna a go with my PDF data extraction, but I think I‘ll stay on Gemini. It does a very good job.
Gemini is still my top choice within production software for typical data extraction from unstructured data. Gemini Flash Lite feels like a cheat code for speed, and it's really cheap. Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.
Speaking from experience here, flash lite models have amazing price, speed, and perform far above their size, but are susceptible to very bad instruction following and recall when either complexity or context size inch up. They’ll just forget to apply your instructions to portions of the input, and repeat parts of the input that should be returned verbatim as direct quotes but with subtle changes (breaking urls, for example).
Re: GPT 5.6 Sol is the best "vision" model OpenAI ever released
#127Earlier quoted context omitted.
Hi, I’m the author of this blog. It depends on how strong of a model you need, but in general, Qwen is easily the best among the Chinese models right now. Over the last two weeks, Qwen released two new models. Qwen3.8-Max is totally insane, but it’s only available through the Alibaba Cloud API. I wrote a similar blog covering Qwen3.8-Max: [ https://blog.roboflow.com/qwen3-8-max/ ]( https://blog.roboflow.com/qwen3-8-m…
Googles local gemma models which target roughly the same parameter count range, are known for being a lot better at vision tasks than qwen, no idea if 3.8 has changed that though
Really? Gemma4-31B should be better than Qwen3.8-27B? I'm happy to test that.
Re: GPT 5.6 Sol is the best "vision" model OpenAI ever released
#128Vision is still embarrassingly bad. ChatGPT Pro with GPT 5.6-sol: https://chatgpt.com/share/6a834217-ca8c-83e8-a8e8-45d5b8797b... The puzzle: https://activityvillage-files.s3.eu-west-2.amazonaws.com/s3f...
The second answer is far more revealing than the first:
OP:
> do you think you did a good job there
ChatGPT:
> I spent 15 minutes, emitted several fake-sounding “tracing the puzzle” progress updates, and then gave a confident permutation without showing that I had actually followed the lines correctly. It reads much more like I guessed than solved it. The only part I did well was obeying the “no Python or tools” instruction.
My observations:
1) Sarcastic tone suggests pre-prompting, or frequent (and therefore stored in memories) denigration of the model in past conversations. I'm leaning the former - it sounds like it was instructed to read admission of defeat.
2) The part about "no Python or tools" is setting the model up for failure.
I mean, this task is, for a human, basically a game of "simulate a line following robot in your head". Pretty sure a VLM could solve that if it was allowed to do the same thing. Off the top of my head, an algorithm like:
1. Identify start and end points
2. Foreach start point, follow next pixel minimizing angle, until endpoint is reached.
3. Report answer
It's literally what every human facing this task does.
EDIT:
My attempt - same image, prompt altered to allow for code (but still no search/external checks), solved in 1/5th of the time, correctly, and (going by thinking trace summaries that I don't think show up in shared chats), basically the same way I'd approach it, by tracing the lines, coloring them as it goes.
https://chatgpt.com/share/6a834f76-8240-83ed-acff-0c67af399d...
INB4: I know this is now not a pure vision check, but it really doesn't make much sense to diss models for failing to solve tasks explicitly designed to teach humans to externalize computation that's hard to do in their heads (i.e. kids, crayons, coloring paths).
Still, if such things are becoming a benchmark for tool-less evaluation, it's only a matter of time until the models learn - much like humans learn in school - to follow algorithms mentally, essentially emulating an ad-hoc computer in their head.
Re: GPT 5.6 Sol is the best "vision" model OpenAI ever released
#129Earlier quoted context omitted.
Hi! I’m the author of this blog. I’m evaluating these VLMs to figure out which ones are good enough to auto-annotate my data, so I can fine-tune my detector. I wrote a bit more about this here: https://x.com/skalskip92/status/2080334344061694429?s=20
Did you evaluate any that could be self-hosted (or at least ow models), if so which one is the best you seen?
Take a look here: https://playground.roboflow.com/evals. We have few ~30B.
Re: GPT 5.6 Sol is the best "vision" model OpenAI ever released
#130I actually favor Qwen3.8 and run it locally + use the Token-Plan on AlibabaCloud, when I need faster results. Kind of favor it over GPT5.6 Sol. Also it seems to be more capable, need to test more, but I think it's at least getting on par and it's fully open-source and open-weights. Here's some benchmarks: https://benchlm.ai/compare/gpt-5-6-sol-vs-qwen3-8-max https://qwen.ai/blog?id=qwen3.8#full-benchmark-table (incre…
huh, why am I being shadow banned?
Does YC have similar problems like those at wikipedia/reddit? (wikipedia-editor-wars, or reddit-mod-wars)