511–520 of 1001 posts
5.3 codex https://openai.com/index/introducing-gpt-5-3-codex/ crushes with a 77.3% in Terminal Bench. The shortest lived lead in less than 35 minutes. What a time to be alive!
claude swe-bench is 80.8 and codex is 56.8 Seems like 4.6 is still all-around better?
The bicycle frame is a bit wonky but the pelican itself is great: https://gist.github.com/simonw/a6806ce41b4c721e240a4548ecdbe...
Earlier quoted context omitted.
Also explains why Claude Code is a React app outputting to a Terminal. (Seriously.)
I did some debugging on this today. The results are... sobering. Memory comparison of AI coding CLIs (single session, idle): | Tool | Footprint | Peak | Language | |-------------|-----------|--------|---------------| | Codex | 15 MB | 15 MB | Rust | | OpenCode | 130 MB | 130 MB | Go | | Claude Code | 360 MB | 746 MB | Node.js/React | That's a 24x to 50x difference for tools that do the same thing: send text to an API…
This is just regular tech debt that happens from building something to $1bn in revenue as fast as you possibly can, optimize later.
They're optimizing now. I'm sure they'll have it under control in no time.
CC is an incredible product (so is codex but I use CC more). Yes, lately it's gotten bloated, but the value it provides makes it bearable until they fix it in short time.
Earlier quoted context omitted.
One aspect of this is that apparently most people can't draw a bicycle much better than this: they get the elements of the frame wrong, mess up the geometry, etc.
There's a research paper from the University of Liverpool, published in 2006 where researchers asked people to draw bicycles from memory and how people overestimate their understanding of basic things. It was a very fun and short read. It's called "The science of cycology: Failures to understand how everyday objects work" by Rebecca Lawson. https://link.springer.com/content/pdf/10.3758/bf03195929.pdf
Earlier quoted context omitted.
Gemini-pro-preview is on ollama and requires h100 which is ~$15-30k. Google are charging $3 a million tokens. Supposedly its capable of generating between 1 and 12 million tokens an hour. Which is profitable. but not by much.
What do you mean it's on ollama and requires h100? As a proprietary google model, it runs on their own hardware, not nvidia.
https://ollama.com/library/gemini-3-pro-preview
You can run it on your own infra. Anthropic and openAI are running off nvidia, so are meta(well supposedly they had custom silicon, I'm not sure if its capable of running big models) and mistral.
however if google really are running their own inference hardware, then that means the cost is different (developing silicon is not cheap...) as you say.
I see
Earlier quoted context omitted.
Dumb question. Can these benchmarks be trusted when the model performance tends to vary depending on the hours and load on OpenAI’s servers? How do I know I’m not getting a severe penalty for chatting at the wrong time. Or even, are the models best after launch then slowly eroded away at to more economical settings after the hype wears off?
On benchmarks GPT 5.2 was roughly equivalent to Opus 4.5 but most people who've used both for SWE stuff would say that Opus 4.5 is/was noticeably better
I think two things are getting conflated in this discussion. First: marginal inference cost vs total business profitability. It’s very plausible (and increasingly likely) that OpenAI/Anthropic are profitable on a per-token marginal basis, especially given how cheap equivalent open-weight inference has become. Third-party providers are effectively price-discovering the floor for inference. Second: model lifecycle econ…
Dario said this in a podcast somewhere. The models themselves have so far been profitable if you look at their lifetime costs and revenue. Annual profitability just isn't a very good lens for AI companies because costs all land in one year and the revenue all comes in the next. Prolific AI haters like Ed Zitron make this mistake all the time.
Just tested the new Opus 4.6 (1M context) on a fun needle-in-a-haystack challenge: finding every spell in all Harry Potter books. All 7 books come to ~1.75M tokens, so they don't quite fit yet. (At this rate of progress, mid-April should do it ) For now you can fit the first 4 books (~733K tokens). Results: Opus 4.6 found 49 out of 50 officially documented spells across those 4 books. The only miss was "Slugulus Eruc…
If I use Opus 4.6 with Extended Thinking (Web Search disabled, no books attached), it answers with 130 spells.