471–480 of 944 posts
So far, it seems like this is the hierarchy o1 > GPT-4o > o3-mini > o1-mini > GPT-4o-mini o3 mini system card: https://cdn.openai.com/o3-mini-system-card.pdf
If this is the hierarchy, why does 4o score so much higher than o1 on LLM Arena? Worrisome for OpenAI that Gemini's mini/flash reasoning model outscores both o1 and 4o handily.
Well, o3-mini-high just successfully found the root cause of a seg fault that o1 missed: mistakenly using _mm512_store_si512 for an unaligned store that should have been _mm512_storeu_si512.
Earlier quoted context omitted.
There's only two kinds of software, prototype and obsolete. I was taught that last millennium.
That's not true. Is Google Maps a prototype or obsolete?
I just pushed a new release of my LLM CLI tool with support for the new model and the reasoning_effort option: https://llm.datasette.io/en/stable/changelog.html#v0-21 Example usage: llm -m o3-mini 'write a poem about a pirate and a walrus' \ -o reasoning_effort high Output (comparing that with the default reasoning effort): https://github.com/simonw/llm/issues/728#issuecomment-262832... (If anyone has a better demo p…
A reasoning model is not meant for writing poetry. It's not very useful to evaluate it on such tasks.
Hopefully this is a big improvement from o1. o1 has been very disappointing after spending sufficient time with Claude Sonnet 3.5. It's like it actively tries to gaslight me and thinks it knows more than I do. It's too stubborn and confidently goes off in tangents, suggesting big changes to parts of the code that aren't the issue. Claude tends to be way better at putting the pieces together in its not-quite-mental-mo…
I've also noticed that with cGPT. That said I often run into a sort of opposite issue with Claude. It's very good at making me feel like a genius. Sometimes I'll suggest trying a specific strategy or trying to define a concept on my own, and Claude enthusiastically agrees and takes us down a 2-3 hour rabbit hole that ends up being quite a waste of time for me to back track out of. I'll then run a post-mortem through…
The truth is these models are very stochastic you have to try new chats whenever you even moderately suspect you're going awry
Earlier quoted context omitted.
No, but the chat divs are all contenteditable.
Oh! That is an excellent solution. I wish it was that easy in every UI.
Earlier quoted context omitted.
That's not true. Is Google Maps a prototype or obsolete?
The website or the database? I'd say the former is obsolete and the latter is still a prototype.
The API pricing is almost exactly double the deepseek ones.
I like deepseek a lot. But they are currently very glitchy. The API service goes up and down a lot. Maybe they'll sort that out soon.
I used o3-mini to summarize this thread so far. Here's the result: https://gist.github.com/simonw/09e5922be0cbb85894cf05e6d75ae... For 18,936 input, 2,905 output it cost 3.3612 cents. Here's the script I used to do it: https://til.simonwillison.net/llms/claude-hacker-news-themes...
3.3612 cents (I guess USD cents) is expensive!
O3 Mini is probably not a very large model and OpenAI has layers upon layers of efficiencies, so they must be making an absolute killing charging 3.3 cents for a few seconds of compute
Earlier quoted context omitted.
I think OpenAI really needs to rethink its product naming, especially now that they have a portfolio where there's no such clear hierarchy, but they have a place along different axis (speed, cost, reasoning, capabilities, etc). Your summary attempt e.g. also misses o3-mini vs o3-mini-high. Lots of trade-ofs.
Can't wait for the eventual rename to GPT Core, GPT Plus, GPT Pro, and GPT Pro Max models! I can see it now: > Unlock our industry leading reasoning features by upgrading to the GPT 4 Pro Max plan.