Ask HN: When will we hit a limit on LLM performance?

#1

Post by ksj2114 » Fri, Jun 14, 2024, 12:12 PM UTC

All the AI founders (e.g., Dario Amodei) seem to believe that we're nowhere near the end of seeing performance improvements in LLMs as they are trained on more data (i.e., LLM scaling laws) - at least that's what they say publicly, but they obviously have skin in the game. Curious what knowledgeable people think who are not incentivized to make optimistic public statements?

What I really want to know is, assuming capital / compute is not a constraint, will be continue to see order of magnitude improvements in LLMs, or is there some kind of "technological" limit you think exists?

Re: Ask HN: When will we hit a limit on LLM performance?

#2

Post by henry_pulver » Fri, Jun 14, 2024, 12:36 PM UTC

As far as I (ex-ML researcher) know, the main technological case that LLM performance will hit a limit is due to the amount of text data available to train on is limited. The ways these scaling laws work is they require 10x or 100x quantity of data to see major improvements.

This isn't necessarily going to limit it though. It's possible there are clever approaches to leverage much more data. This could either be through AI-generated data, other modalities (e.g. video) or another approach altogether.

This is quite a good accessible post on both sides of this discussion: https://www.dwarkeshpatel.com/p/will-scaling-work

Re: Ask HN: When will we hit a limit on LLM performance?

#3

Post by smartician » Fri, Jun 14, 2024, 5:52 PM UTC

Research seems to suggest we need exponential training data volume increases to see meaningful performance gains: https://arxiv.org/abs/2404.04125

Personally I think we've already hit a ceiling.

Re: Ask HN: When will we hit a limit on LLM performance?

#4

Post by dang » Fri, Jun 14, 2024, 9:38 PM UTC

henry_pulver wrote: post #2

As far as I (ex-ML researcher) know, the main technological case that LLM performance will hit a limit is due to the amount of text data available to train on is limited. The ways these scaling laws work is they require 10x or 100x quantity of data to see major improvements. This isn't necessarily going to limit it though. It's possible there are clever approaches to leverage much more data. This could either be thro…

Discussed here:

Will scaling work? - https://news.ycombinator.com/item?id=38781484 - Dec 2023 (283 comments)

Re: Ask HN: When will we hit a limit on LLM performance?

#5

Post by p1esk » Sat, Jun 15, 2024, 1:45 AM UTC

smartician wrote: post #3

Research seems to suggest we need exponential training data volume increases to see meaningful performance gains: https://arxiv.org/abs/2404.04125 Personally I think we've already hit a ceiling.

We have pretty much infinite training data available on YouTube. We can scale by many orders of magnitude before we run out of data. Why do you think we hit a ceiling?