Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5 is behind schedule

wsj.com

Post reply on HN
Print view View original

231–240 of 1001 posts

Re: GPT-5 is behind schedule

#231

Post by mnk47 »

Animats wrote: post #123

"Orion’s problems signaled to some at OpenAI that the more-is-more strategy, which had driven much of its earlier success, was running out of steam." So LLMs finally hit the wall. For a long time, more data, bigger models, and more compute to drive them worked. But that's apparently not enough any more. Now someone has to have a new idea. There's plenty of money available if someone has one. The current level of LLM…

> So LLMs finally hit the wall

Not really. Throwing a bunch of unfiltered garbage at the pretraining dataset, throwing in RLHF of questionable quality during post-training, and other current hacks - none of that was expected to last forever. There is so much low-hanging fruit that OpenAI left untouched and I'm sure they're still experimenting with the best pre-training and post-training setups.

One thing researchers are seeing is resistance to post-training alignment in larger models, but that's almost the opposite of a wall, they're figuring it out as well.

> Now someone has to have a new idea

OpenAI already has a few, namely the o* series in which they discovered a way to bake Chain of Thought into the model via RL. Now we have reasoning models that destroy benchmarks that they previously couldn't touch.

Anthropic has a post-training technique, RLAIF, which supplants RLHF,and it works amazingly well. Combined with countless other tricks we don't know about in their training pipeline, they've managed to squeeze so much performance out of Sonnet 3.5 for general tasks.

Gemini is showing a lot of promise with their new Flash 2.0 and Flash 2.0-Thinking models. They're the first models to beat Sonnet at many benchmarks since April. The new Gemini Pro (or Ultra? whatever they call it now) is probably coming out in January.

> The current level of LLM would be far more useful if someone could get a conservative confidence metric out of the internals of the model. This technology desperately needs to output "Don't know" or "Not sure about this, but ..." when appropriate.

You would probably enjoy this talk [0], it's by an independent researcher who IIRC is a former employee of Deepmind or some other lab. They're exploring this exact idea. It's actually not hard to tell when a model is "confused" (just look at the probability distribution of likely tokens), the challenge is in steering the model to either get back to the right track or give up and say "you know what, idk"

[0] https://www.youtube.com/watch?v=4toIHSsZs1c

Re: GPT-5 is behind schedule

#232

Post by dkobia »

t_serpico wrote: post #208

One fundamental challenge to me is that if each training run because more and more expensive, the time it takes it to learn what works/doesn't work widens. Half a billion dollars for training a model is already nuts, but if it takes 100 iterations to perfect it, you've cumulatively spent 50 billion dollars... Smaller models may actually be where rapid innovation continues simply because of tighter feedback loops. O3…

AGI is the Sisyphean task of our age. We’ll push this boulder up the mountain because we have to, even if it kills us.

Re: GPT-5 is behind schedule

#233

Post by threeseed »

ramesh31 wrote: post #220
t_serpico wrote: post #208

One fundamental challenge to me is that if each training run because more and more expensive, the time it takes it to learn what works/doesn't work widens. Half a billion dollars for training a model is already nuts, but if it takes 100 iterations to perfect it, you've cumulatively spent 50 billion dollars... Smaller models may actually be where rapid innovation continues simply because of tighter feedback loops. O3…

But if the scaling law holds true, more dollars should at some point translate into AGI, which is priceless. We haven't reached the limits yet of that hypothesis.

a) There is evidence e.g. private data deals that we are starting to hit the limitations of what data is available.

b) There is no evidence that LLMs are the roadmap to AGI.

c) Continued investment hinges on their being a large enough cohort of startups that can leverage LLMs to generate outsized returns. There is no evidence yet this is the case.

Re: GPT-5 is behind schedule

#234

Post by sebzim4500 »

Y_Y wrote: post #222
ynniv wrote: post #177

Earlier quoted context omitted.

Is it "eerie"? LeCun has been talking about it for some time, and may also be OpenAI's rumored q-star, mentioned shortly after Noam Brown (diplomacybot) joining OpenAI. You can't hill climb tokens, but you can climb manifolds.

> You can't hill climb tokens, but you can climb manifolds. Could you explain this a bit please?

I imagine he means that when you reason in latent space the final answer is a smooth function of the parameters, which means you can use gradient descent to directly optimize the model to produce a desired final output without knowing the correct reasoning steps to get there.

When you reason in token space (like everyone is doing now) you are executing nonlinear functions when you sample after each token, so you have to use some kind of reinforcement learning algorithm to learn the weights.

Re: GPT-5 is behind schedule

#236

Post by mrbungie »

famouswaffles wrote: post #167
peepeepoopoo98 wrote: post #145

Earlier quoted context omitted.

O3 has demonstrated that OpenAI needs 1,000,000% more inference time compute to score 50% higher on benchmarks. If O3-High costs about $350k an hour to operate, that would mean making O4 score 50% higher would cost $3.5B (!!!) an hour. That scaling wall.

Not really. o3-low compute still stomps the benchmarks and isn't anywhere that expensive and o3-mini seems better than o1 while being cheaper. Combine that with the fact that LLM inference has reduced orders of magnitudes in cost the last few years and hampering over the inference costs of a new release seems a bit silly.

It is still not economical: in Arc at least 20 usd for task vs ~3 usd for a human (avg mturker) for the same perf.

Re: GPT-5 is behind schedule

#237

Post by BoorishBears »

imiric wrote: post #197
BoorishBears wrote: post #184

Earlier quoted context omitted.

I think my biggest pet peeve is when someone shares an insight which is unmistakably based on intuition, inference, critical thinking, etc (all mental faculties we are allowed to use to come to conclusions in the face of information asymmetry btw) ...and then gets hit deadpan with the good old "Source?", like it's some sort of gotcha. I think people have started to confuse "making logical conclusions without perfect…

It's reasonable to ask for sources when an opinion is phrased as a fact, as GGP did. I don't see how you got that it was _unmistakably_ an opinion from that comment. There is no way to deduce by intuition alone that GPT-5 == GPT-4o. So either that person has some information the rest of us aren't privy to, or it's an opinion phrased as a fact. In either case, it deserves clarification.

That was clearly phrased like a fact, which may or may not be correct. If it had been phrased like an opinion we wouldn't be having this conversation...

The problem is once you believe their fact is wrong, just say "I think you're wrong ". Innocently asking for a source as if you're still on the fence is just performative and leads to these conversations where both sides just end up talking past each other:

A source for one underpinning of the incorrect fact comes up, then "well but that only proves X part of it, can you prove Y" and so on.

tl;dr I just find the quality of discourse is much higher when people are direct.

Re: GPT-5 is behind schedule

#238

Post by simonw »

benatkin wrote: post #228
simonw wrote: post #223

Earlier quoted context omitted.

If this was a random blog post I wouldn't nitpick, but this is the Wall Street Journal.

The thing is that I think it could be an optimal way of saying it. Should we not put it into context of making a particular LLM? Why count three versions of three LLMs? They made it hard to choose the one that makes up for not having GPT 1. GPT 3.5 and Codex are both good candidates. And of course calling GPT 4 the third and fifth could be considered as well.

"OpenAI's fourth family of LLMs" or "fourth generation of LLMs" would work for me.

Re: GPT-5 is behind schedule

#239

Post by [deleted] »

A_D_E_P_T wrote: post #68
h_tbob wrote: post #35

Earlier quoted context omitted.

That so weird, it’s seems like everybody here prefers Claude. I’ve been using Claude and openai in copilot and I find even 4o seems to understand the problem better. O1 definitely seems to get it right more for me.

They're both okay for coding, though for my use cases (which are niche and involve quite a lot of mathematics and formal logic) o1/o1-Pro is better. It seems to have a better native grasp of mathematical concepts, and it can even answer very difficult questions from vague inputs, e.g.: https://chatgpt.com/share/676020cb-8574-8005-8b83-4bed5b13e1...

[deleted]

Re: GPT-5 is behind schedule

#240

Post by mrbungie »

threeseed wrote: post #190
synapsomorphy wrote: post #135

Earlier quoted context omitted.

The new idea is already here and it's reasoning / chain of thought. Anecdotally Claude is pretty good at knowing the bounds of its knowledge.

Anecdotally Claude is just as bad as every other LLM. Step into more niche areas e.g. I am trying to use it with Scala macros and at least 90% of the time it is giving code that either (a) fails to compile or (b) is just complete gibberish. And at no point ever has it said it didn't know something.

Yep, get into any sufficiently deep niche (i.e. actually almost any non-trivial app) and the LLM magic fades off.

Yeah sure you can make a pong clone in html/js and that's mainly because there the internet is full of pong clone demos. Ask how to constraint a statsmodels lineal model in some non-standard way? It will gaslight how it is possible and make you loss time in the process.

Post reply on HN