Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New LLM optimization technique slashes memory costs

venturebeat.com

Post reply on HN
Print view View original

81–90 of 227 posts

Re: New LLM optimization technique slashes memory costs

#81

Post by yard2010 »

yunohn wrote: post #76
_aavaa_ wrote: post #48

Earlier quoted context omitted.

Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Others are trying to restart an existing reactor at three mile island (a thing that has never been done before, and likely won't be done now since the reactor was shu…

Look, I agree that nuclear is difficult, but Google and Microsoft have publicly committed to those projects you’re mentioning. I don’t understand your dismissive tone that all of it is hogwash? This is one of those HN armchair comments.

Google and Microsoft won't do anything that doesn't translate to money. These days are over.

Re: New LLM optimization technique slashes memory costs

#82

Post by baq »

solarkraft wrote: post #54

It’s mind bogglingly crazy that language models rivaling ones that used to require huge GPUs with a ton of VRAM to run now run on my upper-mid-range laptop from 4 years ago. At usable speed. Crazy. I didn’t expect capable language models to be practical/possible to run loyally, much less on hardware I already have.

You have a sota multi-modal LLM running in your head at 20W, shared with best in class sensor package and top performing robotics control unit.

There’s soooo much more to optimize.

Re: New LLM optimization technique slashes memory costs

#83

Post by baq »

K0balt wrote: post #25
guerrilla wrote: post #16

Earlier quoted context omitted.

As far as I know, finance is not all in. I see Goldman Sachs doing experiments, for example, but it doesn't feel like they're convinced yet.

Finance is basically all of the reasons not to use (generative, LLM based) AI , all in one vertical. The poster child of determinism.

Finance is all in on reading 10-Ks and generating summaries. If you have decisions in mind, I’ll be referring to IBM 1979 slide until an HR LLM fires me.

Re: New LLM optimization technique slashes memory costs

#85

Post by lm28469 »

odyssey7 wrote: post #36

Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?

It's called the rebound effect, at no point in modern history efficiency reduced our energy needs, we just use the extra energy to either run more of the same thing or run other things

Re: New LLM optimization technique slashes memory costs

#86

Post by benatkin »

vlovich123 wrote: post #5

Wonder how this compares with Microsoft's HeadKV paper [1] which claims a 98% percent reduction in memory while retaining 97% of the performance. [1] https://arxiv.org/html/2410.19258v3

Seems like a different thing. That paper appears to be memory reduction in caching while article appears to be memory reduction in content.

Re: New LLM optimization technique slashes memory costs

#87

Post by fecal_henge »

baq wrote: post #82
solarkraft wrote: post #54

It’s mind bogglingly crazy that language models rivaling ones that used to require huge GPUs with a ton of VRAM to run now run on my upper-mid-range laptop from 4 years ago. At usable speed. Crazy. I didn’t expect capable language models to be practical/possible to run loyally, much less on hardware I already have.

You have a sota multi-modal LLM running in your head at 20W, shared with best in class sensor package and top performing robotics control unit. There’s soooo much more to optimize.

But can it know love?

Re: New LLM optimization technique slashes memory costs

#88

Post by prox »

solarkraft wrote: post #54

It’s mind bogglingly crazy that language models rivaling ones that used to require huge GPUs with a ton of VRAM to run now run on my upper-mid-range laptop from 4 years ago. At usable speed. Crazy. I didn’t expect capable language models to be practical/possible to run loyally, much less on hardware I already have.

I might have a go at installing one, what is a good source or install at the moment?

Re: New LLM optimization technique slashes memory costs

#89

Post by ndr »

odyssey7 wrote: post #36

Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?

Wirth's law: software is getting slower more rapidly than hardware is becoming faster.

I think there's the energy parallel: Software is becoming more energy-hungry faster than algorithms are becoming efficient.

So we'll still need the energy.

Re: New LLM optimization technique slashes memory costs

#90

Post by wint3rmute »

prox wrote: post #88
solarkraft wrote: post #54

It’s mind bogglingly crazy that language models rivaling ones that used to require huge GPUs with a ton of VRAM to run now run on my upper-mid-range laptop from 4 years ago. At usable speed. Crazy. I didn’t expect capable language models to be practical/possible to run loyally, much less on hardware I already have.

I might have a go at installing one, what is a good source or install at the moment?

Ollama was the easiest way to set up local LLMs for me.

https://ollama.com/

Post reply on HN