Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New LLM optimization technique slashes memory costs

venturebeat.com

Post reply on HN
Print view View original

91–100 of 227 posts

Re: New LLM optimization technique slashes memory costs

#91

Post by portaouflop »

cma wrote: post #75
_aavaa_ wrote: post #48

Earlier quoted context omitted.

Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Others are trying to restart an existing reactor at three mile island (a thing that has never been done before, and likely won't be done now since the reactor was shu…

> Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Not building new, but I think Microsoft paying to restart a reactor at Three Mile Island for their datacenter is much more significant than you make the deals sound…

They say it’s going to be online in 2028.

Are you willing to bet that they won’t have 3 mile island operational by 2030?

Re: New LLM optimization technique slashes memory costs

#92

Post by spacemanspiff01 »

prox wrote: post #88
solarkraft wrote: post #54

It’s mind bogglingly crazy that language models rivaling ones that used to require huge GPUs with a ton of VRAM to run now run on my upper-mid-range laptop from 4 years ago. At usable speed. Crazy. I didn’t expect capable language models to be practical/possible to run loyally, much less on hardware I already have.

I might have a go at installing one, what is a good source or install at the moment?

lmstudio is very easy if you are running on local desktop.

Re: New LLM optimization technique slashes memory costs

#93

Post by portaouflop »

yunohn wrote: post #76
_aavaa_ wrote: post #48

Earlier quoted context omitted.

Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Others are trying to restart an existing reactor at three mile island (a thing that has never been done before, and likely won't be done now since the reactor was shu…

Look, I agree that nuclear is difficult, but Google and Microsoft have publicly committed to those projects you’re mentioning. I don’t understand your dismissive tone that all of it is hogwash? This is one of those HN armchair comments.

Microsoft also committed publicly to prioritise security. And Google says they prioritise privacy of their users above all else.

I pity the fool that believes anything these corporations put out publicly.

Actions matter, words are wind

Re: New LLM optimization technique slashes memory costs

#94

Post by vlovich123 »

benatkin wrote: post #86
vlovich123 wrote: post #5

Wonder how this compares with Microsoft's HeadKV paper [1] which claims a 98% percent reduction in memory while retaining 97% of the performance. [1] https://arxiv.org/html/2410.19258v3

Seems like a different thing. That paper appears to be memory reduction in caching while article appears to be memory reduction in content.

They’re both exploring the same space of optimizing the memory needed by the KV cache which is essentially another name for the context window (no one elides the KV cache as otherwise you’re doing N^2 math to do attention). They’re exploring different approaches to achieve the same goal and they may be both possible to apply simultaneously to reduce the attention mechanism to almost 0 memory usage which would be really cool, but I’m curious how they compare against each other individually.

Re: New LLM optimization technique slashes memory costs

#95

Post by nwmcsween »

bongodongobob wrote: post #63
asdev wrote: post #28

Earlier quoted context omitted.

no one is successfully using LLMs for anything other than customer service related things and text generation(coding, writing)

Rubbish. I built a pipeline to handle document classification that successfully took care of ~70TB of mostly unstructured and unorganized data, by myself, in a couple weeks, with no data engineering background whatsoever. This was quite literally impossible a couple years ago. The amount of work that saved was massive and is going to save us a shit ton of money on storage costs. Decades worth of invoices and random P…

I mean considering I did document classification back in 2010 using tesseract, I wouldn't say it was impossible.

Re: New LLM optimization technique slashes memory costs

#96

Post by benatkin »

vlovich123 wrote: post #94
benatkin wrote: post #86

Earlier quoted context omitted.

Seems like a different thing. That paper appears to be memory reduction in caching while article appears to be memory reduction in content.

They’re both exploring the same space of optimizing the memory needed by the KV cache which is essentially another name for the context window (no one elides the KV cache as otherwise you’re doing N^2 math to do attention). They’re exploring different approaches to achieve the same goal and they may be both possible to apply simultaneously to reduce the attention mechanism to almost 0 memory usage which would be real…

That sounds like a stretch to me. If not, I’m impressed how the articles can describe such similar things in such different terms.

Re: New LLM optimization technique slashes memory costs

#97

Post by thom »

odyssey7 wrote: post #36

Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?

What if it's a big hoax and we create a better world for nothing?

Re: New LLM optimization technique slashes memory costs

#98

Post by bonoboTP »

owenpalmer wrote: post #67
odyssey7 wrote: post #36

Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?

No. This is a classic case of Jevon's paradox. Increased efficiency in resource use can lead to increased consumption of that resource, rather than decreased consumption. Example: 1. To decrease total gas consumption, more fuel efficient vehicles are invented. 2. Instead of using less gas, people drive more miles . They take longer road trips, commute farther for work, and more people can now afford to drive. 3. This…

There's no paradox in that. People became more capable and can afford to do more.

Re: New LLM optimization technique slashes memory costs

#99

Post by bonoboTP »

baq wrote: post #80
owenpalmer wrote: post #67

Earlier quoted context omitted.

No. This is a classic case of Jevon's paradox. Increased efficiency in resource use can lead to increased consumption of that resource, rather than decreased consumption. Example: 1. To decrease total gas consumption, more fuel efficient vehicles are invented. 2. Instead of using less gas, people drive more miles . They take longer road trips, commute farther for work, and more people can now afford to drive. 3. This…

See also https://en.m.wikipedia.org/wiki/Induced_demand

How do you argue that demand was induced as opposed to existing demand served?

Re: New LLM optimization technique slashes memory costs

#100

Post by oblio »

vlovich123 wrote: post #94
benatkin wrote: post #86

Earlier quoted context omitted.

Seems like a different thing. That paper appears to be memory reduction in caching while article appears to be memory reduction in content.

They’re both exploring the same space of optimizing the memory needed by the KV cache which is essentially another name for the context window (no one elides the KV cache as otherwise you’re doing N^2 math to do attention). They’re exploring different approaches to achieve the same goal and they may be both possible to apply simultaneously to reduce the attention mechanism to almost 0 memory usage which would be real…

> they may be both possible to apply simultaneously to reduce the attention mechanism to almost 0 memory usage which would be really cool

https://matt.might.net/articles/why-infinite-or-guaranteed-f...

Post reply on HN