Any good leaderboard where all the very latest models are compared?
OpenAI o3 and o4-mini
321–330 of 527 posts
Re: OpenAI o3 and o4-mini
#322So at this point OpenAI has 6 reasoning models, 4 flagship chat models, and 7 cost optimized models. So that's 17 models in total and that's not even counting their older models and more specialized ones. Compare this with Anthropic that has 7 models in total and 2 main ones that they promote. This is just getting to be a bit much, seems like they are trying to cover for the fact that they haven't actually done much.…
Think for 30 seconds about why they might in good faith do what they do.
Do you use any of them? Are you a developer? Just because a model is non-deterministic it doesn't mean developers don't want some level of consistency, whether it be about capabilities, cost, latency, call structure etc.
Re: OpenAI o3 and o4-mini
#323Here's a summary of this conversation so far, generated using o3 after 306 comments. This time I ran it like so:
llm install llm-openai-plugin
llm install llm-hacker-news
llm -m openai/o3 -f hn:43707719 -s 'Summarize the themes of the opinions expressed here.
For each theme, output a markdown header.
Include direct "quotations" (with author attribution) where appropriate.
You MUST quote directly from users when crediting them, with double quotes.
Fix HTML entities. Output markdown. Go long. Include a section of quotes that illustrate opinions uncommon in the rest of the piece'
Then again with o4-mini using the exact same content (hence the hash ID for -f):
llm -m openai/o4-mini \
-f f16158f09f76ab5cb80febad60a6e9d5b96050bfcf97e972a8898c4006cbd544 \
-s 'Summarize the themes of the opinions expressed here.
For each theme, output a markdown header.
Include direct "quotations" (with author attribution) where appropriate.
You MUST quote directly from users when crediting them, with double quotes.
Fix HTML entities. Output markdown. Go long. Include a section of quotes that illustrate opinions uncommon in the rest of the piece'
Cost 2,684 input, 2,681 output (of which 1,088 reasoning tokens) = 1.4749 cents
The above uses these two plugins: https://github.com/simonw/llm-openai-plugin and https://github.com/simonw/llm-hacker-news - taking advantage of new -f "fragments" feature I released last week: https://simonwillison.net/2025/Apr/7/long-context-llm/
Re: OpenAI o3 and o4-mini
#324So at this point OpenAI has 6 reasoning models, 4 flagship chat models, and 7 cost optimized models. So that's 17 models in total and that's not even counting their older models and more specialized ones. Compare this with Anthropic that has 7 models in total and 2 main ones that they promote. This is just getting to be a bit much, seems like they are trying to cover for the fact that they haven't actually done much.…
> This is just getting to be a bit much, seems like they are trying to cover for the fact that they haven't actually done much. All these models feel like they took the exact same base model, tweaked a few things and released it as an entirely new model
OpenAI's progress lately:
2024 December - first reasoning model (official release)
2025 February - deep search
2025 March - true multi-modal image generation
2025 April - reasoning model with tools
Re: OpenAI o3 and o4-mini
#325Earlier quoted context omitted.
Meanwhile even the highest ranked models can’t do simple logic tasks. GothamChess on YouTube did some tests where he played against a bunch of the best models and every single one of them failed spectacularly. They’d happily lose a queen to take a pawn. They failed to understand how pieces are even allowed to move, hallucinated the existence of new pieces, repeatedly declared checkmate when it wasn’t, etc. I tried it…
I'm not sure why people are expecting a language model to be great at chess. Remember they are trained on text, which is not the best medium for representing things like a chess board. They are also "general models", with limited training on pretty much everything apart from human language. An Alpha Star type model would wipe the floor at chess.
This misses the point. LLMs will do things like move a knight by a single square as if it were a pawn. Chess is an extremely well understood game, and the rules about how things move is almost certainly well-represented in the training data.
These models cannot even make legal chess moves. That’s incredibly basic logic, and it shows how LLMs are still completely incapable of reasoning or understanding. Many kinds of task are never going to be possible for LLMs unless that changes. Programming is one of those tasks.
Re: OpenAI o3 and o4-mini
#326Earlier quoted context omitted.
Gemini 2.5 Pro for every single task was the meta until this release. Will have to reassess now.
Huh. I use Gemini 2.0 Flash for many things because it's several times faster than 2.5 Pro.
Yes, this one is addictive for its speed and I like how Google was clever and also offered it in a powerful reasoning edition. This helps offset deficiencies from being smaller while still being cheap. I also find it quite sufficient for my kind of coding. I only pull out 2.5 Pro on larger and complex code bases that I think might need deeper domain specific knowledge beyond the coding itself.
Re: OpenAI o3 and o4-mini
#327Earlier quoted context omitted.
Have any of the models been deprecated? It seems like a deprecation plan and definition of timelines would be extraordinarily helpful. I have not seen any sort of "If you're using X.122, upgrade to X.123, before 202X. If you're using X.120, upgrade to anything before April 2026, because the model will no longer be available on that date." ... Like all operating systems and hardware manufacturers have been doing for d…
Yep, we have a page of announced API deprecations here: https://platform.openai.com/docs/deprecations It's got all deprecations, ordered by date of announcement, alongside shutdown dates and recommended replacements. Note that we use the term deprecated to mean slated for shutdown, and shutdown to mean when it's actually shut down. In general, we try to minimize developer pain by supporting models for as long as we r…
On that page I don't see any mention of o3-mini. Is o3-mini a legacy model now which is slated to be deprecated later on?
Re: OpenAI o3 and o4-mini
#328Re: OpenAI o3 and o4-mini
#329Please just give me a best value and a highest performance model.
Re: OpenAI o3 and o4-mini
#330Earlier quoted context omitted.
Meanwhile even the highest ranked models can’t do simple logic tasks. GothamChess on YouTube did some tests where he played against a bunch of the best models and every single one of them failed spectacularly. They’d happily lose a queen to take a pawn. They failed to understand how pieces are even allowed to move, hallucinated the existence of new pieces, repeatedly declared checkmate when it wasn’t, etc. I tried it…
I'm not sure why people are expecting a language model to be great at chess. Remember they are trained on text, which is not the best medium for representing things like a chess board. They are also "general models", with limited training on pretty much everything apart from human language. An Alpha Star type model would wipe the floor at chess.
> I'm not sure why people are expecting a language model to be great at chess.
Because the conversation is about AGI, and how far away we are from AGI.