Nitpicking but this page is not practical to share has there's no individual url per post (AFAIK) (the # part is not picked up by Slack etc to generate preview).
OpenAI o3-pro
171–180 of 209 posts
Re: OpenAI o3-pro
#172Earlier quoted context omitted.
You may be above average intelligence. Those challenges are like classic IQ tests and I bet have a significant distribution among humans.
No, they've done testing against samples from the general population.
The original ARC-AGI test was much easier than the recent v2.
Re: OpenAI o3-pro
#173Earlier quoted context omitted.
I'm seeing big advances that arent shown in the benchmarks, I can simply build software now that I couldnt build before. The level of complexity that I can manage and deliver is higher.
Okay but this has all to do with the tooling and nothing to do with the models.
I mostly disagree with this.
I have been using 'aider' as my go to coding tool for over a year. It basically works the same way that it always has: you specify all the context and give it a request and that goes to the model without much massaging.
I can see a massive improvement in results with each new model that arrives. I can do so much more with Gemini 2.5 or Claude 4 than I could do with earlier models and the tool has not really changed at all.
I will agree that for the casual user, the tools make a big difference. But if you took the tool of today and paired it with a model from last year, it would go in circles
Re: OpenAI o3-pro
#174I understand that things are moving fast and all, but surely the.. 8? models which are currently available is a bit .. overwhelming for users that just want to get answers to their questions of life? What's the end goal with having so many models available?
I'd like one to do my test use case: Port unix-sed from c to java with a full test suite and all options supported. Somewhere between "it answers questions of life" and "it beats PhDs at math questions", I'd like to see one LLM take this, IMO, rather "pure" language task and succeeed. It is complicated, but it isn't complex. It's string operations with a deep but not that deep expression system and flag set. It is we…
The massive real market here is enterprises that need to rewrite legacy code to modern platforms, retaining the business logic as-is but modernising the style.
.NET Framework 4.x to .NET 10, Python 2 to 3, Java 8 to , etc...
The advantage the LLMs have here is that staying within the same programming language and its paradigm is dramatically simpler than converting a "procedural" language like C to an object-oriented language like Java that has a wildly different standard library.
Re: OpenAI o3-pro
#175Earlier quoted context omitted.
Thanks for your input, very appreciated. Just in case you didn’t mean Claude Code, it’s really good in my experience and mostly stable. If something fails, it just retries and I don’t notice it much. Its autonomous discovery and tool use is really good and I‘m relying more and more on it.
For the Claude issues, I'm referring to the claude.ai frontend. While I use some Codex, Aider, and other agentic tools, I found Claude Code to be not to my taste - for my uses it tended burn a lot of tokens and gave relatively mediocre results, but I know it works well for others, so YMMV.
If you're happy with your current tools that's good, but if not, and if you haven't tried Claude Code recently, you might give it a retry. I'm not sure what all they've been changing, but it burns a lot fewer tokens for me on tasks now than it did when I first started using it, with better results.
Re: OpenAI o3-pro
#176Re: OpenAI o3-pro
#177Earlier quoted context omitted.
No, this doesn't seem to be correct, although confusion regarding model names is understandable. o4-mini-high is the label on chatgpt.com for what in the API is called o4-mini with reasoning={"effort": "high"}. Whereas o4-mini on chatgpt.com is the same thing as reasoning={"effort": "medium"} in the API. o3 can also be run via the API with reasoning={"effort": "high"}. o3-pro is different than o3 with high reasoning.…
OpenAI started strong in the naming department (ChatGPT, DALL-E) then fell off so hard since.
It's arguable ChatGPT is not such a great name, either. The general public has no idea what GPT means and will often swap the letters around. It does benefit from being unique, however.
Re: OpenAI o3-pro
#178Earlier quoted context omitted.
There are humans who cannot do arc agi though so how does an LLM not doing it mean that LLMs don’t have general intelligence? LLMs have obviously reached the point where they are smarter than almost every person alive, better at maths, physics, biology, English, foreign languages, etc. But because they can’t solve this honestly weird visual/spatial reasoning test they aren’t intelligent? That must mean most humans on…
> LLMs have obviously reached the point where they are smarter than almost every person alive, better at maths, physics, biology, English, foreign languages, etc. I dont think memorizing stuff is the same as being smart. https://en.wikipedia.org/wiki/Chinese_room > But because they can’t solve this honestly weird visual/spatial reasoning test they aren’t intelligent? Yes. Being intelligent is about recognizing patter…
> I dont think memorizing stuff is the same as being smart. https://en.wikipedia.org/wiki/Chinese_room
I agree. The problem I have with the Chinese Room thought experiment is: just as the human who mechanically reading books to answer questions they don't understands does not themselves know Chinese, likewise no neuron in the human brain knows how the brain works.
The intelligence, such as it is, is found in the process that generated the structure — of the translation books in the Chinese room, of the connectome in our brains, and of the weights in an LLM.
What comes out of that process is an artefact of intelligence, and that artefact can translate Chinese or whatever.
Because all current AI take a huge number of examples to learn anything, I think it's fair to say they're not particularly intelligent — but likewise, they can to an extent make up for being stupid by being stupid very very quickly.
But: this definition of intelligence doesn't really fit "can solve novel puzzle", as there's a lot of room for getting good at that my memorising lot of things that puzzle-creators tend to do.
And any mind (biological or synthetic) must learn patterns before getting started: the problem of induction* is that no finite number of examples is ever guaranteed to be sufficient to predict the next item in a sequence, there is always an infinite set of other possible solutions in general (though in reality bounded by 2^n, where n = the number of bits required to express the universe in any given state).
I suspect, but cannot prove, that biological intelligence learns from fewer examples for a related reason, that our brains have been given a bias by evolution towards certain priors from which "common sense" answers tend to follow. And "common sense" is often wrong, c.f. Aristotelian physics (never mind Newtonian) instead of QM/GR.
Re: OpenAI o3-pro
#179Gemini still, for me, feels like the king for speed and accuracy.
Gemini 2.5 Pro is incredible - both in coding and text review.
DeepSeek isn't bad either (especially given its age now), and Claude is great for coding and tool use but too damn expensive.
Re: OpenAI o3-pro
#180I'm really hoping GPT5 is a larger jump in metrics than the last several releases we've seen like Claude3.5 - Claude4 or o3-mini-high to o3-pro. Although I will preface that with the fact I've been building agents for about a year now and despite the benchmarks only showing slight improvement, I have seen that each new generation feels actively better at exactly the same tasks I gave the previous generation. It would…
I remember the saying that from 90% to 99% is a 10x increase in accuracy, but 99% to 99.999% is a 1000x increase in accuracy. Even though it's a large10% increase first then only a 0.999% increase.
What's the required computation power for those extra 9s? Is it linear, poly, or exponential?
Imo we got to the current state by harnessing GPUs for a 10-20x boost over CPUs. Well, and cloud parallelization, which is ?100x?
ASIC is probably another 10x.
But the training data may need to vastly expand, and that data isn't going to 10x. It's probably going to degrade.