Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5 is behind schedule

wsj.com

Post reply on HN
Print view View original

131–140 of 1001 posts

Re: GPT-5 is behind schedule

#131

Post by bobxmax »

omega3 wrote: post #23
diego_sandoval wrote: post #12

Earlier quoted context omitted.

Reminds me of this Louis CK joke: I was on an airplane and there was high-speed Internet on the airplane. That's the newest thing that I know exists. And I'm sitting on the plane and they go, open up your laptop, you can go on the Internet. And it's fast, and I'm watching YouTube clips. It's amazing. I'm on an airplane! And then it breaks down. And they apologize, the Internet's not working. And the guy next to me go…

People who say „it’s bullshit” are the ones that push the technological advance forward.

That's really not true.

Re: GPT-5 is behind schedule

#132

Post by bobxmax »

leonheld wrote: post #128
bobxmax wrote: post #121

Earlier quoted context omitted.

We are close. Language models and large vision models have transformed robotics. It just takes some time to get hardware up and running.

> have transformed robotics Did they? Where? Seriously, I genuinely want to know who is employing these techniques.

All frontier labs are now employing LVMs or LLMs. But that's my point is you won't see the fruits of it this early.

Re: GPT-5 is behind schedule

#133

Post by viraptor »

noman-land wrote: post #69
ericskiff wrote: post #25

What we can reasonably assume from statements made by insiders: They want a 10x improvement from scaling and a 10x improvement from data and algorithmic changes The sources of public data are essentially tapped Algorithmic changes will be an unknown to us until they release, but from published research this remains a steady source of improvement Scaling seems to stall if data is limited So with all of that taken toge…

I completely don't understand the use for synthetic data. What good it's it to train a model basically on itself?

This is a good read for some examples https://arxiv.org/abs/2203.14465

> This technique, the "Self-Taught Reasoner" (STaR), relies on a simple loop: generate rationales to answer many questions, prompted with a few rationale examples; if the generated answers are wrong, try again to generate a rationale given the correct answer; fine-tune on all the rationales that ultimately yielded correct answers; repeat. We show that STaR significantly improves performance on multiple datasets compared to a model fine-tuned to directly predict final answers

But there are a few others. In general good data is good data. We're definitely learning more about how to produce good synthetic version.

Re: GPT-5 is behind schedule

#134

Post by ldjkfkdsjnv »

A_D_E_P_T wrote: post #2

Counterpoint: o1-Pro is insanely good -- subjectively, it's as far above GPT4 as GPT4 was above 3. It's almost too good. Use it properly for an extended period of time, and one begins to worry about the future of one's children and the utility of their schooling. o3, by all accounts, is better still. Seems to me that things are progressing quickly enough.

It basically solves all bugs/programming challenges i throw at it, given i give it the right data

Re: GPT-5 is behind schedule

#135

Post by synapsomorphy »

Animats wrote: post #123

"Orion’s problems signaled to some at OpenAI that the more-is-more strategy, which had driven much of its earlier success, was running out of steam." So LLMs finally hit the wall. For a long time, more data, bigger models, and more compute to drive them worked. But that's apparently not enough any more. Now someone has to have a new idea. There's plenty of money available if someone has one. The current level of LLM…

The new idea is already here and it's reasoning / chain of thought.

Anecdotally Claude is pretty good at knowing the bounds of its knowledge.

Re: GPT-5 is behind schedule

#136

Post by layer8 »

ynniv wrote: post #60
dartos wrote: post #50

Earlier quoted context omitted.

I’m curious how, if at all, the plan to get around compounding bias in synthetic data generated by models trained in synthetic data.

Everyone's obsessed with new training tokens... It doesn't need to be more knowledgeable, it just needs to practice more. Ask any student: practice is synthetic data.

And who will tell the model whether its practice results are correct or not? Students practice against external evaluators, it’s not a self-contained system.

Re: GPT-5 is behind schedule

#137

Post by fragmede »

leonheld wrote: post #128
bobxmax wrote: post #121

Earlier quoted context omitted.

We are close. Language models and large vision models have transformed robotics. It just takes some time to get hardware up and running.

> have transformed robotics Did they? Where? Seriously, I genuinely want to know who is employing these techniques.

https://www.figure.ai/

specifically their speech demo video (which is, of course, a demo video)

https://youtu.be/Sq1QZB5baNw

https://www.1x.tech/neo and

https://www.unitree.com/h1/

are undoubtedly using such models.

It's an area of active research, eg

https://www.physicalintelligence.company/blog/pi0

https://wholebody-b1.github.io/

https://ok-robot.github.io/

https://mobile-aloha.github.io/

Re: GPT-5 is behind schedule

#138

Post by whoisthemachine »

Animats wrote: post #123

"Orion’s problems signaled to some at OpenAI that the more-is-more strategy, which had driven much of its earlier success, was running out of steam." So LLMs finally hit the wall. For a long time, more data, bigger models, and more compute to drive them worked. But that's apparently not enough any more. Now someone has to have a new idea. There's plenty of money available if someone has one. The current level of LLM…

Unfortunately, the best they can do is "This is my confidence on what someone would say given the prior context".

Re: GPT-5 is behind schedule

#139

Post by briga »

Animats wrote: post #123

"Orion’s problems signaled to some at OpenAI that the more-is-more strategy, which had driven much of its earlier success, was running out of steam." So LLMs finally hit the wall. For a long time, more data, bigger models, and more compute to drive them worked. But that's apparently not enough any more. Now someone has to have a new idea. There's plenty of money available if someone has one. The current level of LLM…

What wall? Not a week has gone by in recent years without an LLM breaking new benchmarks. There is little evidence to suggest it will all come to a halt in 2025.

Re: GPT-5 is behind schedule

#140

Post by simonw »

Animats wrote: post #123

"Orion’s problems signaled to some at OpenAI that the more-is-more strategy, which had driven much of its earlier success, was running out of steam." So LLMs finally hit the wall. For a long time, more data, bigger models, and more compute to drive them worked. But that's apparently not enough any more. Now someone has to have a new idea. There's plenty of money available if someone has one. The current level of LLM…

The new idea is inference-time scaling, as seen in o1 (and o3 and Qwen's QwQ and DeepSeek's DeepSeek-R1-Lite-Preview and Google's gemini-2.0-flash-thinking-exp).

I suggest reading these two pieces about that:

- https://www.aisnakeoil.com/p/is-ai-progress-slowing-down - best explanation I've seen of inference scaling anywhere

- https://arcprize.org/blog/oai-o3-pub-breakthrough - François Chollet's deep dive into o3

I've been tracking it on this tag on my blog: https://simonwillison.net/tags/inference-scaling/

Post reply on HN