Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o3 and o4-mini

openai.com

Post reply on HN
Print view View original

441–450 of 527 posts

Re: OpenAI o3 and o4-mini

#441

Post by departed »

rsanheim wrote: post #179

`ETOOMANYMODELS` Is there a reputable, non-blogspam site that offers a 'cheat sheet' of sorts for what models to use, in particular for development? Not just openAI, but across the main cloud offerings and feasible local models? I know there are the benchmarks, and directories like huggingface, and you can get a 'feel' for things by scanning threads here or other forums. I'm thinking more of something that provides u…

LMArena might have some of the information you are looking for. It offers rankings of LLM models across main cloud offerings, and I feel that its evaluation method, human prompting and voting, is closer to real-world use case and less prone to data contamination than benchmarks.

https://lmarena.ai/

In the "Leaderboard">"Language" tab, it lists the top models in various categories such as overall, coding, math, and creative writing.

In the "Leaderboard">"Price Analysis" tab, it shows a chart comparing models by cost per million tokens.

In the "Prompt-to-Leaderboard" tab, there is even an LLM to help you find LLMs -- you enter a prompt, and it will find the top models for your particular prompt.

Re: OpenAI o3 and o4-mini

#442

Post by simonw »

CaptainFever wrote: post #433
brokencode wrote: post #64

Earlier quoted context omitted.

I personally like being able to choose because I understand the tradeoffs and want to choose the best one for what I’m asking. So I hope this doesn’t go away. But I agree that they probably need some kind of basic mode to make things easier for the average person. The basic mode should decide automatically what model to use and hide this from the user.

Would that be considered a Mixture of Experts system?

No, Mixture of Experts is a really confusing term.

It sounds like it means "have a bunch of models, one that's an expert in physics, one that's an expert in health etc and then pick the one that's a best fit for the user's query".

It's not that. The "experts" are each another giant opaque blob of weights. The model is trained to select one of those blobs, but they don't have any form of human-understandable "expertise". It's an optimization that lets you avoid using ALL of the weights for every run through the model, which helps with performance.

https://huggingface.co/blog/moe#what-is-a-mixture-of-experts... is a decent explanation.

Re: OpenAI o3 and o4-mini

#443

Post by M4v3R »

shmerl wrote: post #387
M4v3R wrote: post #298

Ok, I’m a bit underwhelmed. I’ve asked it a fairly technical question, about a very niche topic (Final Fantasy VII reverse engineering): https://chatgpt.com/share/68001766-92c8-8004-908f-fb185b7549... With right knowledge and web searches one can answer this question in a matter of minutes at most. The model fumbled around modding forums and other sites and did manage to find some good information but then started to…

How would it ever know the answer it found is true and correct though? It could as well just repeat some existing false answer that you didn't yet find on your own. That's not much better than hallucinating it, since you can't verify its truth without finding it independently anyway.

I would be ok with having an answer and an explanation of how it got the answer with a list of sources. And it does just that - the only problem is that both the answer and the explanation are fabrications after you double check the sources.

Re: OpenAI o3 and o4-mini

#444

Post by jonahx »

famouswaffles wrote: post #426
jonahx wrote: post #374

Earlier quoted context omitted.

Thanks for the reply. I am not sure the vision is the failing point here, but logic. I routinely try to get these models to solve difficult puzzles or coding challenges (the kind that a good undergrad math major could probably solve, but that most would struggle with). They fail almost always. Even with help. For example, JaneStreet monthly puzzles. Surprisingly, the new o3 was able to solve this months (previous mod…

>what it can do incredible -- but I don't believe the phrase AGI should even be mentioned until we are seeing solutions to problems that most professional mathematicians would struggle with, including solutions to unsolved problems. But Why ? Why should Artificial General Intelligence preclude things a good chunk of humans wouldn't be able to do ? Are those guys no longer General Intelligences ? I'm not saying this d…

Sure, there's no authority who stamps the official definition.

I'll make my case. To me, if you look at how the phrase is usually used -- "when humans have achieved AGI...", etc -- it evokes a science fiction turning point that implies superhuman performance in more or less every intellectual task. It's general, after all. I think of Hal or the movie Her. It's not "Artifical General Just-Like-Most-People-You-Know Intelligence". Though we are not there yet, either, if you consider the full spectrum of human abilities.

Few things would demonstrate general superhuman reasoning ability more definitively than machines producing new, useful, influential math results at a faster rate than people. With that achieved, you would expect it could start writing fiction and screenplays and comedy as well as people too (it's still very far imo), but maybe not, maybe those skills develop at different paces, and I still wouldn't want to call it AGI. But I think truly conquering mathematics would get me there.

Re: OpenAI o3 and o4-mini

#445

Post by M4v3R »

int_19h wrote: post #386
M4v3R wrote: post #298

Ok, I’m a bit underwhelmed. I’ve asked it a fairly technical question, about a very niche topic (Final Fantasy VII reverse engineering): https://chatgpt.com/share/68001766-92c8-8004-908f-fb185b7549... With right knowledge and web searches one can answer this question in a matter of minutes at most. The model fumbled around modding forums and other sites and did manage to find some good information but then started to…

Compare to Gemini Pro 2.5: https://g.co/gemini/share/c8fb1c9795e4 Of note, the final step in the CoT is: > Formulate Conclusion: Since a definitive list or count isn't readily available through standard web searches, the best approach is to: state that an exact count is difficult to ascertain from readily available online sources without direct analysis of game files ... avoid giving a specific number, as none was re…

I like this answer. It does mention the correct, definitive way of getting the information I want (extracting the char.lgp data file) and so even though it gave up it pushes you in the right direction, whereas o3/o4 just make up stuff.

Re: OpenAI o3 and o4-mini

#446

Post by Seattle3503 »

_fat_santa wrote: post #58

So at this point OpenAI has 6 reasoning models, 4 flagship chat models, and 7 cost optimized models. So that's 17 models in total and that's not even counting their older models and more specialized ones. Compare this with Anthropic that has 7 models in total and 2 main ones that they promote. This is just getting to be a bit much, seems like they are trying to cover for the fact that they haven't actually done much.…

This seems like a perfect use case for "agentic" AI. OpenAI can enrich the context window with the strengths and weakness of each model, and when a user prompts for something the model can say "Hey, I'm gonna switch to another model that is better at answering this sort of question." and the user can accept or reject.

Re: OpenAI o3 and o4-mini

#447

Post by extr »

SkyPuncher wrote: post #419
werdnapk wrote: post #382

Earlier quoted context omitted.

I've used AI with "niche" programming questions and it's always a total let down. I truly don't understand this "vibe coding" movement unless everyone is building todo apps.

There's a bit of a skill to it. Good architecture plans help. Telling it where in an existing code base it can find things to pattern match against is also fantastic. I'll often end up with a task that looks something like this: * Implement Foo with a relation to FooBar. * Foo should have X, Y, Z features * We have an existing pattern for Fidget in BigFidget. Look at that for implementation * Make sure you account fo…

Yeah this is a great summary of what I do as well and I find it very effective. I think of hands-off AI coding like you're directing a movie. You have a rough image of what "good" looks like in your head, and you're trying to articulate it with enough detail to all the stagehands and actors such that they can realize the vision. The models can always get there with enough coaching, traditionally the question is if that's worth the trouble versus just doing it yourself.

Increasingly I find that AI at this point is good enough I am rarely stepping in to "do it myself".

Re: OpenAI o3 and o4-mini

#448

Post by glial »

NiloCK wrote: post #434
osigurdson wrote: post #153

I have a very basic / stupid "Turing test" which is just to write a base 62 converter in C#. I would think this exact thing would be in github somewhere (thus in the weights) but has always failed for me in the past (non-scientific / didn't try every single model). Using o4-mini-high, it actually did produce a working implementation after a bit of prompting. So yeah, today, this test passed which is cool.

I could be misinterpreting your claim here, but I'll point out that LLM weights don't literally encode the entirety of the training data set.

I guess you could consider it a lossy encoding.

Re: OpenAI o3 and o4-mini

#449

Post by motorest »

werdnapk wrote: post #382
M4v3R wrote: post #298

Ok, I’m a bit underwhelmed. I’ve asked it a fairly technical question, about a very niche topic (Final Fantasy VII reverse engineering): https://chatgpt.com/share/68001766-92c8-8004-908f-fb185b7549... With right knowledge and web searches one can answer this question in a matter of minutes at most. The model fumbled around modding forums and other sites and did manage to find some good information but then started to…

I've used AI with "niche" programming questions and it's always a total let down. I truly don't understand this "vibe coding" movement unless everyone is building todo apps.

> I've used AI with "niche" programming questions and it's always a total let down.

That's perfectly fine. It just means you tried without putting in any effort and failed to get results that were aligned with your expectations.

I'm also disappointed when I can't dunk or hit >50% of my 3pt shots, but then again I never played basketball competitively

> I truly don't understand this "vibe coding" movement unless everyone is building todo apps.

Yeah, I also don't understand the NBA. Every single one of those players show themselves dunking and jumping over cars and having almost perfect percentages in 3pt shots during practice, whereas I can barely get off my chair. The problem is certainly basketball.

Re: OpenAI o3 and o4-mini

#450

Post by Foobar8568 »

mhitza wrote: post #416
hatefulmoron wrote: post #389

Earlier quoted context omitted.

It's incredible when I ask Claude 3.7 a question about Typescript/Python and it can generate hundreds of lines of code that are pretty on point (it's usually not exactly correct on first prompt, but it's coherent). I've recently been asking questions about Dafny and Lean -- it's frustrating that it will completely make up syntax and features that don't exist, but still speak to me with the same confidence as when it'…

Don't need to ho that esoteric. Seen them make stuff up pretty often for more common functional programming languages like Haskell and OCaml.

Well all LLM are fairly bad for react native as soon as you look at more than hello world type of things.

I got stuck with different LLM until I checked the official documentation, yeah spouting nonsense from 2y+ removed features I suppose or just making up stuff.

Post reply on HN