Wake me up when the full o3 is out.
[deleted]
OpenAI O3-Mini
461–470 of 944 posts
Re: OpenAI O3-Mini
#462The API pricing is almost exactly double the deepseek ones.
I like deepseek a lot. But they are currently very glitchy. The API service goes up and down a lot. Maybe they'll sort that out soon.
Re: OpenAI O3-Mini
#463> Testers preferred o3-mini's responses to o1-mini 56% of the time I hope by this they don't mean me, when I'm asked 'which of these two responses do you prefer'. They're both 2,000 words, and I asked a question because I have something to do. I'm not reading them both ; I'm usually just selecting the one that answered first. That prompt is pointless. Perhaps as evidenced by the essentially 50% response rate: it's a…
I don't think they make it clear: I wonder if they mean testers prefer o3 mini 56% of the time when they express an opinion, or overall? Some percentage of people don't choose; if that number is 10% and they aren't excluded, that means 56% of the time people prefer o3 mini, 34% of the time people prefer o1 mini, and 10% of the time people don't choose. I'm not sure I think it would be reasonable to present the data that way, but it seems possible.
Re: OpenAI O3-Mini
#464Earlier quoted context omitted.
A reasoning model is not meant for writing poetry. It's not very useful to evaluate it on such tasks.
It's not clear that writing poetry is a bad use case. Reasoning models seem to actually do pretty well with creative writing and poetry. Deepseek's R1, for example, has much better poem structure than the underlying V3, and writers are saying R1 was the first model where they actually felt like it was a useful writing companion. R1 seems to think at length about word choice, correcting structure, pentameter, and so o…
Indeed. I would assume that a reasoning model would do far better at things like actually maintaining meter or rhyme scheme, something that models (even with good attention mechanisms) generally do very poorly at.
Re: OpenAI O3-Mini
#465Earlier quoted context omitted.
Sure you can. "Reasoning" is ultimately an implementation detail, and the only thing that matters for capabilities is results, not process.
By "reasoning" I meant the fact that o*(-mini) does "chain-of-thought", in other words, it prompts itself to "reason" before responding to you, whereas GPT-4o(-mini) just directly responds to your prompt. Thus, it is not appropriate to compare o*(-mini) and GPT-4o(-mini) unless you implement "chain-of-thought" for GPT-4o(-mini) and compare that with o*(-mini). See also: https://docs.anthropic.com/en/docs/build-with-c…
That's like saying you can't compare a sedan to a truck.
Sure you can.
Even though one is more appropriate for certain tasks than the other.
Re: OpenAI O3-Mini
#466Earlier quoted context omitted.
Deepseek is the state of the art right now in terms of performance and output. It's really fast. The way it "explains" how it's thinking is remarkable.
DeepSeek is great because: 1) you can run the model locally, 2) the research was openly shared, and 3) the reasoning tokens are open. It is not, in my experience, state of the art. In all of my side by side comparisons thus far in real world applications between DeepSeek V3 and R1 vs 4o and o1, the latter has always performed better. OpenAI's models are also more consistent, glitching out maybe one in 10,000, whereas…
I watched it complete pretty complicated tasks like "write a snake game in Python" and "write Tetris in Python" successfully. And the way it did it, with showing all the internal steps, I've never seen before.
Watch here. https://www.youtube.com/watch?v=by9PUlqtJlM
Re: OpenAI O3-Mini
#467> Testers preferred o3-mini's responses to o1-mini 56% of the time I hope by this they don't mean me, when I'm asked 'which of these two responses do you prefer'. They're both 2,000 words, and I asked a question because I have something to do. I'm not reading them both ; I'm usually just selecting the one that answered first. That prompt is pointless. Perhaps as evidenced by the essentially 50% response rate: it's a…
This is just a way to prove, statistically, that one model is better than another as part of its validation. It's not collected from normal people using ChatGPT, you don't ever get shown two responses from different models at once.
Re: OpenAI O3-Mini
#468Earlier quoted context omitted.
I'm sorry, but it's over for OpenAI. Some have predicted this; including me back in November[1] when I wrote "o1 is a revolution in accounting, not capability" which although tongue-in-cheek, has so far turned out to be correct. I'm only waiting to see what Google, Facebook et al. will accomplish now that R1-Zero result is out the bag. The nerve, the cheek of this hysterical o3-mini release—insisting to hide the COT…
You don't need to wait for Google. Their Jan 21 checkpoint for their fast reasoning model is available on AIStudio. It shows full reasoning traces. It's very good, much faster than R1, and although they haven't released pricing, based on flash it's going to be quite cheap.
Sure, their 01-21 reasoning model is really good, but there's no pricing for it!
I care mostly about batching in Vertex AI, which is 17-30x times cheaper than competition (whether you use prompt caching or not) while allowing for audio, video, and arbitrary document filetype inputs; unfortunately Gemini 1.5 Pro/Flash have remained the two so-called "stable" options that are available there. I can appreciate Google's experimental models for all I can, but I cannot take them seriously until they allow me to have my sweet, sweet batches.
Re: OpenAI O3-Mini
#469> Testers preferred o3-mini's responses to o1-mini 56% of the time I hope by this they don't mean me, when I'm asked 'which of these two responses do you prefer'. They're both 2,000 words, and I asked a question because I have something to do. I'm not reading them both ; I'm usually just selecting the one that answered first. That prompt is pointless. Perhaps as evidenced by the essentially 50% response rate: it's a…
This is just a way to prove, statistically, that one model is better than another as part of its validation. It's not collected from normal people using ChatGPT, you don't ever get shown two responses from different models at once.
Wait what? I get shown this with ChatGPT maybe 5% of the time
Re: OpenAI O3-Mini
#470Earlier quoted context omitted.
The “o” models get the order right. DeepSeek also gets the order right. It doesn’t show on the share link. But it actually outputs the list correctly from the built in Python interpreter. For some things, ChatGPT 4o will automatically use its Python runtime
That some models get it right is irrelevant. In general, if your instructions require computation , it's safer to assume it won't get it right and will hallucinate.
The reasoning models all do pretty good at math.
Have you tried them?
This is something I threw together with o3-mini
https://chatgpt.com/share/679d5305-5f04-8010-b5c4-61c31e79b2...
ChatGPT 4o doesn’t even try to do the math internally and uses its built in Python interpreter. (The [_>] link is to the Python code)
https://chatgpt.com/share/679d54fe-0104-8010-8f1e-9796a08cf9...
DeepSeek handles the same problem just as well using the reasoning technique.
Of course ChatGPT 4o went completely off the rails without using its Python interpreter
https://chatgpt.com/share/679d5692-96a0-8010-8624-b1eb091270...
(The break down that it got right was using Python even though I told it not to)