Can someone please share the logic behind their version naming convention?
OpenAI O3-Mini
421–430 of 944 posts
Re: OpenAI O3-Mini
#422I switched an agent from Sonnet V2 to o3-mini (default medium mode) and got strangely poor results: only calling 1 tool at a time despite being asked to call multiple, not actually doing any work, and reporting that it did things it didn't
Re: OpenAI O3-Mini
#423Earlier quoted context omitted.
Because you're not testing "will a user click the left or right button" (for which asking a thousand users to click a button would be a pretty good estimation), you're testing "which response is preferred". If 10% of people just click based on how fast the response was because they don't want to read both outputs, your p-value for the latter hypothesis will be atrocious, no matter how large the sample is.
> If 10% of people just click based on how fast the response was Couldn't this be considered a form of preference? Whether it's the type of preference OpenAI was testing for, or the type of preference you care about, is another matter.
Sure, it could be, you can define "preference" as basically anything, but it just loses its meaning if you do that. I think most people would think "56% prefer this product" means "when well-informed, 56% of users would rather have this product than the other".
Re: OpenAI O3-Mini
#424I’ll take the China Deluxe instead, actually. I’ve been incredibly pleased with DeepSeek this past week. Wonderful product, I love seeing its brain when it’s thinking.
Using R1 with Perplexity has impressed me in a way that none of the previous models have, and I can't even figure out if it's actually R1, seems likely that its a 70B-llama distillation since that's what AWS offers on Bedrock but from what I can find Perplexity does have their own H100 cluster through Amazon so it's feasible they could be hosting the real thing? But I feel like they would brag about that achievement…
> seems likely that its a 70B-llama distillation since that's what AWS offers on Bedrock
I think you misread something. AWS mainly offers the full size model on Bedrock: https://aws.amazon.com/blogs/aws/deepseek-r1-models-now-avai...
They talk about how to import the distilled models and deploy those if you want, but AWS does not appear to be officially supporting those.
Re: OpenAI O3-Mini
#425Earlier quoted context omitted.
Yes, I am assuming they evaluated the models in good faith, understand how to design a basic user study, and therefore when they ran a study intended to compare the response quality between two different models, they showed the raters both fully-formed responses at the same time, regardless of the actual latency of each model.
I would recommend you read the comment that started this thread then, because that's the context we're talking about: https://news.ycombinator.com/item?id=42891294
I did read that comment. I don't think that person is saying they were part of the study that OpenAI used to evaluate the models. They would probably know if they had gotten paid to evaluate LLM responses.
But I'm glad you pointed that out, I now suspect that is responsible for a large part of the disagreement between "huh? a statistically significant blind evaluation is a statistically significant blind evaluation" vs "oh, this was obviously a terrible study" repliers is due to different interpretations of that post. Thanks. I genuinely didn't consider the alternative interpretation before.
Re: OpenAI O3-Mini
#426Earlier quoted context omitted.
It's like AWS SKU naming (`c5d.metal`, `p5.48xlarge`, etc.), except non-technical consumers are expected to understand it.
Those are not names but hashes used to look up the specs.
I was thinking we might treat model names analogously, but their specs can be moving targets.
Re: OpenAI O3-Mini
#427Earlier quoted context omitted.
R1 or R1-Distill? They are not the same thing. I think DeepSeek made a mistake releasing them at the same time and calling them all R1. Full R1 solves this prompt easily for me.
I used R1 hosted at NVidia here: https://build.nvidia.com/deepseek-ai/deepseek-r1/modelcard
Same. With the recommended settings, it got it right. I regenerated a bunch of times, and it did suggest Cathy once or twice.
R1 70b also got it right just as many times for me.
Re: OpenAI O3-Mini
#428I used o3-mini to summarize this thread so far. Here's the result: https://gist.github.com/simonw/09e5922be0cbb85894cf05e6d75ae... For 18,936 input, 2,905 output it cost 3.3612 cents. Here's the script I used to do it: https://til.simonwillison.net/llms/claude-hacker-news-themes...
Thanks for sharing this! And no apparent self-awareness! OpenAI has come a long way from the Sydney days: https://answers.microsoft.com/en-us/bing/forum/all/this-ai-c...
Re: OpenAI O3-Mini
#429Not really impressed by the answers I just got.
Re: OpenAI O3-Mini
#430Earlier quoted context omitted.
Tier 3 here and already see it on Limits page, so maybe the wait won't be long.
Yep, I got an email about o3-mini in the API an hour ago.
I apparently got one at the same time too, but I missed it distracted by this HN thread :). Not only I got o3-mini (which I already noticed on the Limits page), but they also gave me access to o1 now! I'm Tier 3; until yesterday, o1 was still Tier 5 (IIRC).
Thanks OpenAI! Nice gift and a neat distraction from DeepSeek-R1 - which I still can't use directly, because their API stopped working moments after I topped up my credits and generated an API key, and is still down for me... :/.