Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On the slow death of scaling

papers.ssrn.com

Post reply on HN
Print view View original

11–20 of 34 posts

Re: On the slow death of scaling

#11

Post by gdiamos »

It was an interesting read Sara, thanks for sharing it.

I especially agree with your point that scaling laws really killed open research. That's a shame and I personally think we could benefit from more research.

I originally didn't like calling them scaling laws.

In addition to the law part seeming a bit much, I've found that researchers often overemphasize the scale part. If scaling is predictable, then you don't need to do most experiments at very large scale. However, that doesn't seem to stop researchers from starting there.

Once you find something good, and you understand how it scales, then you can pour system resources into it. So I originally thought it would encourage research. I find it sad that it seems to have had the opposite effect.

Re: On the slow death of scaling

#12

Post by charcircuit »

>the acceptance that there are emergent properties which appear out of nowhere is another way of saying our scaling laws don’t actually equip us to know what is coming.

Is this actually accepted? Ever since [0], I thought people recognized that they don't appear out of nowhere.

[0] https://arxiv.org/pdf/2304.15004

Re: On the slow death of scaling

#13

Post by wiz21c »

FTA:

"One thing is certain, is the less reliable gains from compute makes our purview as computer scientists interesting again. We can now stray from the beaten path of boring, predictable gains from throwing compute at the problem."

Isn't Ilya Sutskever who said some months ago that we were going back to research ?

Re: On the slow death of scaling

#16

Post by Haaargio »

Its not dying slowly right now at all.

Compute is a massive driver for everything ML. From number of experiments you can run in paralle, to how much RL you can try out, how long stuff is running etc.

ML is pushing scaling on dimensions we haven't had before (number of Datacenters, amount of energy we put into them) and ML is currently seen as the holy grail.

But i'm definitly very very curious how this compute and current progress is playing out in the next few years. It could be that we hit a hard ceiling were every single % point becomes tremendesly costly before we hit a % point of benchmark archievements which makes all of that usable daily. OR we will se a significant change to our society.

I do not think its something in between tbh because it def feels like in an expoential progress curve we are currently in.

Re: On the slow death of scaling

#17

Post by drob518 »

I suspect scaling will not die a slow death, but rather slow for a while and then all at once. Further, I think we’re at the knee. We know scaling doesn’t work for resolving the fundamental issues we have with large models at this point. If it did, the latest models would have solved the issues. Now, we’re in the acceptance phase. That’s not technical, it’s human psychology. People who made bold claims and huge promises that things would get better if we just spent a few more billion dollars on data centers and GPUs need to unwind those claims and find a way to save face.

Re: On the slow death of scaling

#20

Post by wrqvrwvq »

SecretDreams wrote: post #5
random3 wrote: post #3

Earlier quoted context omitted.

If anyone believes they're close to a generalization end-game wrt to AI capabilities, it makes no sense to do anything that could impact their advantage, by enabling others to compete. Collaboration makes sense on timeframes that don't imply zero-sum games. Board games like the Settlers of Catan are good examples of the behavior— concretely the start of the game when everyone trades vs the end of the game when if you…

> Collaboration makes sense on timeframes that don't imply zero-sum games. People are fooling themselves if they think AGI will be zero sum. Even if only one group somehow miraculously develops it, there will immediately be fast followers. And, the more likely scenario is more than one group would independently pull it off - if it's even possible.

[dead]
Post reply on HN