Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Failing to Understand the Exponential, Again

julian.ac

Post reply on HN
Print view View original

221–230 of 266 posts

Re: Failing to Understand the Exponential, Again

#221

Post by kryptiskt »

theptip wrote: post #98
crazygringo wrote: post #7

> Given consistent trends of exponential performance improvements over many years and across many industries, it would be extremely surprising if these improvements suddenly stopped. I'm sure people were saying that about commercial airline speeds in the 1970's too. But a lot of technologies turn out to be S-shaped, not purely exponential, because there are limiting factors. With LLM's at the moment, the limiting fac…

> a lot of technologies turn out to be S-shaped, not purely exponential, because there are limiting factors. Yes of course it’s not going to increase exponentially forever. The point is, why predict that the growth rate is going to slow exactly now? What evidence are you going to look at? It’s possible to make informed predictions (eg “Moore’s law can’t get you further than 1nm with silicon due to fundamental physica…

I think progress per dollar spent has actually slowed dramatically over the last three years. The models are better, but AI spending has increased by several orders of magnitude during the same time, from hundreds of millions to hundreds of billions. You can only paper over the lack of fundamental progress by spending on more compute for so long. And even if you manage to keep up the current capex, there certainly isn't enough capital in the world to accelerate spending for very long.

Re: Failing to Understand the Exponential, Again

#222

Post by tempfile »

adammarples wrote: post #161
Inufu wrote: post #111

Earlier quoted context omitted.

Author here. The argument is not that it will keep growing exponentially forever (obviously that is physically impossible), rather that: - given a sustained history of growth along a very predictable trajectory, the highest likelihood short term scenario is continued growth along the same trajectory. Sample a random point on an s-curve and look slightly to the right, what’s the most common direction the curve continu…

The most common part of the S-curve by far is the flat bit before and the flat bit after. We just don't graph it because it's boring. Besides which there is no reason at all to assume that this process will follow that shape. Seems like guesswork backed up by hand waving.

Very much handwaving. The question is not meaningful at all without knowing the parameters of the S-curve. It's like saying "I flipped a coin and saw heads. What's the most likely next flip?"

Re: Failing to Understand the Exponential, Again

#223

Post by ViewTrick1002 »

jsnell wrote: post #213
Someone wrote: post #201

Earlier quoted context omitted.

> Indeed you can't be sure. But on the other hand a bunch of the commentariat has been claiming (with no evidence) that we're at the midpoint of the sigmoid for the last three years. I haven’t followed things closely, but I’ve seen more statements that we may be near the midpoint of a sigmoid than that we are at it. > Thy were wrong. And then you had the AI frontier lab insiders who predicted an accelerating pace of…

I thought the original article included the strongest objective data point on this: recent progress on the METR long task benchmark isn't just on the historical "task length doubling every 7 months" best fit, but is trending above it. A year ago, would you have thought that a pure LLM with no tools could get a gold medal level score in the 2025 IMO finals? I would have thought that was crazy talk. Given the rates of…

Or are the model author’s, i.e the blog author with a vested interest, getting better at optimizing for the test while real world performance aren’t increasing as fast?

Re: Failing to Understand the Exponential, Again

#224

Post by Ravus »

Exponential curves happen when a quantity's growth rate is a linear function of its own value. In practice they're all going to be logistic, but you can ignore that as long as you're far away from the cap of whatever factor limits growth.

So what are the things that could cause "AI growth" (for some suitable definition of it) to be correlated with AI? The plausible ones I see are: - growing AI capabilities spur additional AI capex - AI could be used to develop better AIs

The first one rings true, but is most definitely hitting the limit since US capex into the sector definitely cannot grow 100-fold (and probably cannot grow 4-fold either).

The second one is, to my knowledge, not really a thing.

So unless AI can start improving itself or there is a self-feeding mechanism that I have missed, we're near the logistic fun phase.

Re: Failing to Understand the Exponential, Again

#225

Post by conartist6 »

Wow, an exponential trendline, I guess billions of years of evolution can just give up and go home cause we have rigged the game my friends. At this rate we will create an AI which can do a task 10 years long! And then soon after that 100 years long! And that's that. Humans will be kept as pets because that's all we will be good for QED

Re: Failing to Understand the Exponential, Again

#226

Post by coldtea »

senordevnyc wrote: post #189
coldtea wrote: post #150

Earlier quoted context omitted.

> Except it’s not been five years, it’s been at most three, Why would it be "at most" 3? We had Chat GPT commercially available as private beta API on 2020. It's only the mass public that got 3.5 3 years ago. But those who'd do the noticing as per my argument is not just Joe Public (which could be oblivious), but people already starting in 2020, and includes people working in the space, who worked with LLM and LLM-li…

No, we didn’t. We had the GPT-3 API available in 2020, and approximately no one was using it.

Seems like you missed most of my comment

Re: Failing to Understand the Exponential, Again

#227

Post by ben_w »

crazygringo wrote: post #7

> Given consistent trends of exponential performance improvements over many years and across many industries, it would be extremely surprising if these improvements suddenly stopped. I'm sure people were saying that about commercial airline speeds in the 1970's too. But a lot of technologies turn out to be S-shaped, not purely exponential, because there are limiting factors. With LLM's at the moment, the limiting fac…

>> it would be extremely surprising if these improvements suddenly stopped.

> But a lot of technologies turn out to be S-shaped, not purely exponential, because there are limiting factors.

An S-curve is exactly the opposite of "suddenly" stopping.

It is possible for us to get a sudden stop, due to limiting factors.

For a hypothetical: if Moore's Law had continued until we hit atomic resolution instead of the slowdown as we got close to it, that would have been an example of a sudden stop: can't get transistors smaller than atoms, but yet it would have been possible (with arbitrarily large investments that we didn't have) to halve transistor sizes every 18 months until suddenly we can't.

Now I think about it, the speed of commercial airlines is also an example of a sudden stop: we had to solve sonic booms first before even considering a Concorde replacement.

Re: Failing to Understand the Exponential, Again

#228

Post by skybrian »

breuleux wrote: post #53
crazygringo wrote: post #7

> Given consistent trends of exponential performance improvements over many years and across many industries, it would be extremely surprising if these improvements suddenly stopped. I'm sure people were saying that about commercial airline speeds in the 1970's too. But a lot of technologies turn out to be S-shaped, not purely exponential, because there are limiting factors. With LLM's at the moment, the limiting fac…

> But a lot of technologies turn out to be S-shaped, not purely exponential, because there are limiting factors. I'd argue all of them. Any true exponential eventually gets to a point where no computer can even store its numerical value. It's a physically absurd curve.

Some exponentials are slow enough that it takes decades or centuries, though.

Re: Failing to Understand the Exponential, Again

#229

Post by skybrian »

theptip wrote: post #98
crazygringo wrote: post #7

> Given consistent trends of exponential performance improvements over many years and across many industries, it would be extremely surprising if these improvements suddenly stopped. I'm sure people were saying that about commercial airline speeds in the 1970's too. But a lot of technologies turn out to be S-shaped, not purely exponential, because there are limiting factors. With LLM's at the moment, the limiting fac…

> a lot of technologies turn out to be S-shaped, not purely exponential, because there are limiting factors. Yes of course it’s not going to increase exponentially forever. The point is, why predict that the growth rate is going to slow exactly now? What evidence are you going to look at? It’s possible to make informed predictions (eg “Moore’s law can’t get you further than 1nm with silicon due to fundamental physica…

Yes, nobody knows the future of AI, but sometimes people use curve fitting to try convince themselves or others that they know what’s going to happen.

Re: Failing to Understand the Exponential, Again

#230

Post by tripplyons »

konmok wrote: post #12

As they say, every exponential is a sigmoid in disguise. I think the exponential phase of growth for LLM architectures is drawing to a close, and fundamentally new architectures will be necessary for meaningful advances. I'm also not convinced by the graphs in this article. OpenAI is notoriously deceptive with their graphs, and as Gary Marcus has already noted, that METR study comes with a lot of caveats: [ https://g…

What makes you believe the exponential phase will end soon?
Post reply on HN