for those interested, updated my o3-mini price chart to compare the cost-intelligence frontier with deepseek: https://x.com/swyx/status/1885432031896887335
OpenAI O3-Mini
341–350 of 944 posts
Re: OpenAI O3-Mini
#342I used o3-mini to summarize this thread so far. Here's the result: https://gist.github.com/simonw/09e5922be0cbb85894cf05e6d75ae... For 18,936 input, 2,905 output it cost 3.3612 cents. Here's the script I used to do it: https://til.simonwillison.net/llms/claude-hacker-news-themes...
[deleted]
Re: OpenAI O3-Mini
#343I used o3-mini to summarize this thread so far. Here's the result: https://gist.github.com/simonw/09e5922be0cbb85894cf05e6d75ae... For 18,936 input, 2,905 output it cost 3.3612 cents. Here's the script I used to do it: https://til.simonwillison.net/llms/claude-hacker-news-themes...
I noticed that it thought that GoatInGrey wrote “openai is no longer relevant.” However, they were just quoting a different user (buyucu) who was the person who first wrote that.
Re: OpenAI O3-Mini
#344Earlier quoted context omitted.
There's no moat, and they have to work even harder. Competition is good.
Capex was the theoretical moat, same as TSMC and similar businesses. DeepSeek poked a hole in this theory. OpenAI will need to deliver massive improvements to justify a 1 billion dollar training cost relative to 5 million dollars.
I don't know if you are, but a lot of people are still comparing one Deepseek training run to the entire costs of OpenAI.
The deepseek paper states that the $5mil number doesn't include development costs, only the final training run. And it doesn't include the estimated $1.4billion cost of the infrastructure/chips Deepseek owns.
Most of OpenAI's billion dollar costs is in inference, not training. It takes a lot of compute to serve so many users.
Dario said recently that Claude was in the tens of millions (and that it was a year earlier, so some cost decline is expected), do we have some reason to think OpenAI was so vastly different?
Re: OpenAI O3-Mini
#345> Testers preferred o3-mini's responses to o1-mini 56% of the time I hope by this they don't mean me, when I'm asked 'which of these two responses do you prefer'. They're both 2,000 words, and I asked a question because I have something to do. I'm not reading them both ; I'm usually just selecting the one that answered first. That prompt is pointless. Perhaps as evidenced by the essentially 50% response rate: it's a…
> I'm usually just selecting the one that answered first
Which is why you randomize the order. You aren’t a tester.
56% vs 44% may not be noise. That’s why we have p values. It depends on sample size.
Re: OpenAI O3-Mini
#346Earlier quoted context omitted.
That's such a counter-productive and frankly dumb thing to do. Just don't vote on them.
You have to pick one to continue the chat.
I know for a fact that as of yesterday I did not have to pick one to continue the conversation. It just maximizes the second choice and displayed a 2/2 below the response.
Re: OpenAI O3-Mini
#347Earlier quoted context omitted.
This one is good: https://artificialanalysis.ai/
Looks like this only compares commercial models, and not the ones I can download and actually run locally.
https://livebench.ai/#/
My experience is as follows:
- "Reason" toggle just got enabled for me as a free tier user of ChatGPT's webchat. Apparently this is o3-mini - I have Copilot Pro (offered to me for free), which apparently has o1 too (as well as Sonnet, etc.)
From my experience DeepSeek R1 (webchat) is more expressive, more creative and its writing style is leagues better than OpenAI's models, however it under-performs Sonnet when changing code ("code completion").
Comparison screenshots for prompt "In C++, is a reference to "const C" a "const reference to C"?": https://imgur.com/a/c-is-reference-to-const-c-const-referenc...
tl;dr keep using Claude for code and DeepSeek webchat for technical questions
Re: OpenAI O3-Mini
#348They use the word reasoning a lot in the post. Is this reasoning or statistical prediction?
Re: OpenAI O3-Mini
#349Earlier quoted context omitted.
Those prompts are so irritating and so frequent that I’ve taken to just quickly picking whichever one looks worse at a cursory glance. I’m paying them, they shouldn’t expect high quality work from me.
Have you considered the possibility that your feedback is used to choose what type of response to give to you specifically in the future? I would not consider purposely giving inaccurate feedback for this reason alone.
Spotted the pissed off OpenAI RLHF engineer! Hahahahaha!
Re: OpenAI O3-Mini
#350Earlier quoted context omitted.
If this is the hierarchy, why does 4o score so much higher than o1 on LLM Arena? Worrisome for OpenAI that Gemini's mini/flash reasoning model outscores both o1 and 4o handily.
Is it possible people are voting for speed of responsiveness too?
I suspect people on LLM Arena don't ask complex questions too often, and reasoning models seem to perform worse than simple models when the goal is just casual conversation or retrieving embedded knowledge. Reasoning models probably 'overthink' in such cases. And slower, too.