GPT-5 is behind schedule

Post by tucnak » Sun, Dec 22, 2024, 10:24 PM UTC

sdwr wrote: post #112

ericskiff wrote: post #25

What we can reasonably assume from statements made by insiders: They want a 10x improvement from scaling and a 10x improvement from data and algorithmic changes The sources of public data are essentially tapped Algorithmic changes will be an unknown to us until they release, but from published research this remains a steady source of improvement Scaling seems to stall if data is limited So with all of that taken toge…

Great improvements and all, but they are still no closer (as of 4o regular) to having a system that can be responsible for work. In math problems, it forgets which variable represents what, in coding questions it invents library fns. I was watching a YouTube interview with a "trading floor insider". They said they were really being paid for holding risk. The bank has a position in a market, and it's their ass on the…

Post by jrm4 » Sun, Dec 22, 2024, 10:25 PM UTC

briga wrote: post #139

Animats wrote: post #123

"Orion’s problems signaled to some at OpenAI that the more-is-more strategy, which had driven much of its earlier success, was running out of steam." So LLMs finally hit the wall. For a long time, more data, bigger models, and more compute to drive them worked. But that's apparently not enough any more. Now someone has to have a new idea. There's plenty of money available if someone has one. The current level of LLM…

What wall? Not a week has gone by in recent years without an LLM breaking new benchmarks. There is little evidence to suggest it will all come to a halt in 2025.

Post by peepeepoopoo98 » Sun, Dec 22, 2024, 10:28 PM UTC

briga wrote: post #139

Animats wrote: post #123

"Orion’s problems signaled to some at OpenAI that the more-is-more strategy, which had driven much of its earlier success, was running out of steam." So LLMs finally hit the wall. For a long time, more data, bigger models, and more compute to drive them worked. But that's apparently not enough any more. Now someone has to have a new idea. There's plenty of money available if someone has one. The current level of LLM…

What wall? Not a week has gone by in recent years without an LLM breaking new benchmarks. There is little evidence to suggest it will all come to a halt in 2025.

Post by ijustlovemath » Sun, Dec 22, 2024, 10:30 PM UTC

RedNifre wrote: post #111

psb217 wrote: post #73

Earlier quoted context omitted.

The value of synthetic data relies on having non-zero signal about which generated data is "better" or "worse". In a sense, this what reinforcement learning is about. Ie, generate some data, have that data scored by some evaluator, and then feed the data back into the model with higher weight on the better stuff and lower weight on the worse stuff. The basic loop is: (i) generate synthetic data, (ii) rate synthetic d…

But who rates the synthetic data? If it is humans, I can understand that this is another way to get human knowledge into it, but if it's rated by AI, isn't it just a convoluted way of copying the rating AI's knowledge?

Post by Kuinox » Sun, Dec 22, 2024, 10:31 PM UTC

peepeepoopoo98 wrote: post #145

briga wrote: post #139

Earlier quoted context omitted.

What wall? Not a week has gone by in recent years without an LLM breaking new benchmarks. There is little evidence to suggest it will all come to a halt in 2025.

O3 has demonstrated that OpenAI needs 1,000,000% more inference time compute to score 50% higher on benchmarks. If O3-High costs about $350k an hour to operate, that would mean making O4 score 50% higher would cost $3.5B (!!!) an hour. That scaling wall.