Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New LLM optimization technique slashes memory costs

venturebeat.com

Post reply on HN
Print view View original

61–70 of 227 posts

Re: New LLM optimization technique slashes memory costs

#61

Post by UltraSane »

sbierwagen wrote: post #53
odyssey7 wrote: post #36

Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?

Congrats, you have independently reinvented the Hardware Overhang hypothesis: that early AGI could be very inefficient, undergo several optimization passes, and go from needing a datacenter of compute to, say, a single video game console's worth: https://www.lesswrong.com/posts/75dnjiD8kv2khe9eQ/measuring-... In that scenario, you can go from 0 independent artificial intelligences to tens of millions of them, very qu…

it would seem perfectly reasonable to expect the first AIs to be very unoptimized and if the AIs are any good they will be able to optimize themselves a lot and even help design ASICs to help run them.

Re: New LLM optimization technique slashes memory costs

#62

Post by philipswood »

sbierwagen wrote: post #53
odyssey7 wrote: post #36

Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?

Congrats, you have independently reinvented the Hardware Overhang hypothesis: that early AGI could be very inefficient, undergo several optimization passes, and go from needing a datacenter of compute to, say, a single video game console's worth: https://www.lesswrong.com/posts/75dnjiD8kv2khe9eQ/measuring-... In that scenario, you can go from 0 independent artificial intelligences to tens of millions of them, very qu…

Thanks for sharing. Worth its own submission: https://news.ycombinator.com/newest

Re: New LLM optimization technique slashes memory costs

#63

Post by bongodongobob »

asdev wrote: post #28
mouse_ wrote: post #13

Earlier quoted context omitted.

True. Microsoft's all in, Apple's all in, Nvidia is selling shovels, insurance companies are all in, police & military are all in, education is all in, office management is all in. Who is left to pump line up?

no one is successfully using LLMs for anything other than customer service related things and text generation(coding, writing)

Rubbish. I built a pipeline to handle document classification that successfully took care of ~70TB of mostly unstructured and unorganized data, by myself, in a couple weeks, with no data engineering background whatsoever. This was quite literally impossible a couple years ago. The amount of work that saved was massive and is going to save us a shit ton of money on storage costs. Decades worth of invoices and random PDFs are now siloed properly so we can organize and sort them. This was almost intractable a few years ago.

Re: New LLM optimization technique slashes memory costs

#65

Post by standeven »

_aavaa_ wrote: post #48
odyssey7 wrote: post #36

Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?

Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Others are trying to restart an existing reactor at three mile island (a thing that has never been done before, and likely won't be done now since the reactor was shu…

And unfortunately, gas and coal in the meantime.

https://www.theguardian.com/technology/2024/sep/15/data-cent...

Re: New LLM optimization technique slashes memory costs

#66

Post by eru »

voltagex_ wrote: post #50
eru wrote: post #37

Earlier quoted context omitted.

We are putting lots of optimisation efforts into lots of worthwhile endeavours.

I dunno, software seems to be getting worse, hardware is getting more expensive and both Microsoft and Apple are distracted by AI, not to mention NVIDIA who seem to have bet the farm on Deus Ex Shovel

Hardware is still getting cheaper all the time as far as I can tell.

Though I had thought you were talking about stuff like eg producing more corn on a given piece of land, or making more furniture from less wood or so. Or even just making better batteries and solar cells.

Re: New LLM optimization technique slashes memory costs

#67

Post by owenpalmer »

odyssey7 wrote: post #36

Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?

No. This is a classic case of Jevon's paradox. Increased efficiency in resource use can lead to increased consumption of that resource, rather than decreased consumption.

Example:

1. To decrease total gas consumption, more fuel efficient vehicles are invented.

2. Instead of using less gas, people drive more miles. They take longer road trips, commute farther for work, and more people can now afford to drive.

3. This increased driving leads to higher overall gasoline consumption, despite each car using gas more efficiently.

Re: New LLM optimization technique slashes memory costs

#68

Post by m463 »

odyssey7 wrote: post #36

Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?

Don't you think people will just add better models to meet available memory?

If we run 7B now, why wouldn't we run 700b with memory optimizations?

Re: New LLM optimization technique slashes memory costs

#69

Post by bubaumba »

h0l0cube wrote: post #41
hackernewds wrote: post #34

Earlier quoted context omitted.

Like what?

Oh, I don’t know, how about reducing the search space/accelerating the search speed for potential room temperature superconductors? Or how about the same for viable battery chemistries?

> search speed for potential room temperature superconductors?

and what if it's a dead end?

Re: New LLM optimization technique slashes memory costs

#70

Post by aitchnyu »

_aavaa_ wrote: post #48
odyssey7 wrote: post #36

Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?

Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Others are trying to restart an existing reactor at three mile island (a thing that has never been done before, and likely won't be done now since the reactor was shu…

Could be a good candidate for factobattery. Overbuild the system, run them at full speed at peak solar generation, then underclock them at night.

https://www.moderndescartes.com/essays/factobattery/

Post reply on HN