Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New LLM optimization technique slashes memory costs

venturebeat.com

Post reply on HN
Print view View original

71–80 of 227 posts

Re: New LLM optimization technique slashes memory costs

#71

Post by aitchnyu »

bongodongobob wrote: post #63
asdev wrote: post #28

Earlier quoted context omitted.

no one is successfully using LLMs for anything other than customer service related things and text generation(coding, writing)

Rubbish. I built a pipeline to handle document classification that successfully took care of ~70TB of mostly unstructured and unorganized data, by myself, in a couple weeks, with no data engineering background whatsoever. This was quite literally impossible a couple years ago. The amount of work that saved was massive and is going to save us a shit ton of money on storage costs. Decades worth of invoices and random P…

Could you describe your stack and how its much more effective than two years ago? I heard of printed-table OCR and doc classification years back.

Re: New LLM optimization technique slashes memory costs

#72

Post by BoorishBears »

keyle wrote: post #14
ComputerGuru wrote: post #6

This only decreases memory cost of input context window, not the memory cost to load and run the models.

I agree with you though, the title is misleading.

Title is perfect. Their typical audience probably understands "memory" better than "context window", but then if you've actually deployed these systems it's not difficult to go the other way, from "memory" to "context window" since the context window specifically is known to take additional VRAM over the model itself

Re: New LLM optimization technique slashes memory costs

#73

Post by adelpozo »

_aavaa_ wrote: post #48
odyssey7 wrote: post #36

Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?

Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Others are trying to restart an existing reactor at three mile island (a thing that has never been done before, and likely won't be done now since the reactor was shu…

Or solar in space, which some have already heard of Lumen Orbit https://www.ycombinator.com/companies/lumen-orbit

Re: New LLM optimization technique slashes memory costs

#74

Post by musha68k »

bongodongobob wrote: post #63
asdev wrote: post #28

Earlier quoted context omitted.

no one is successfully using LLMs for anything other than customer service related things and text generation(coding, writing)

Rubbish. I built a pipeline to handle document classification that successfully took care of ~70TB of mostly unstructured and unorganized data, by myself, in a couple weeks, with no data engineering background whatsoever. This was quite literally impossible a couple years ago. The amount of work that saved was massive and is going to save us a shit ton of money on storage costs. Decades worth of invoices and random P…

Very interesting. If I may ask: how are you handling the correctness issue? What's the workflow there if even able to spot a mishap?

Re: New LLM optimization technique slashes memory costs

#75

Post by cma »

_aavaa_ wrote: post #48
odyssey7 wrote: post #36

Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?

Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Others are trying to restart an existing reactor at three mile island (a thing that has never been done before, and likely won't be done now since the reactor was shu…

> Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before.

Not building new, but I think Microsoft paying to restart a reactor at Three Mile Island for their datacenter is much more significant than you make the deals sound:

https://www.theguardian.com/environment/2024/sep/20/three-mi...

Re: New LLM optimization technique slashes memory costs

#76

Post by yunohn »

_aavaa_ wrote: post #48
odyssey7 wrote: post #36

Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?

Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Others are trying to restart an existing reactor at three mile island (a thing that has never been done before, and likely won't be done now since the reactor was shu…

Look, I agree that nuclear is difficult, but Google and Microsoft have publicly committed to those projects you’re mentioning. I don’t understand your dismissive tone that all of it is hogwash? This is one of those HN armchair comments.

Re: New LLM optimization technique slashes memory costs

#77

Post by dx034 »

yunohn wrote: post #76
_aavaa_ wrote: post #48

Earlier quoted context omitted.

Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Others are trying to restart an existing reactor at three mile island (a thing that has never been done before, and likely won't be done now since the reactor was shu…

Look, I agree that nuclear is difficult, but Google and Microsoft have publicly committed to those projects you’re mentioning. I don’t understand your dismissive tone that all of it is hogwash? This is one of those HN armchair comments.

But not just for AI, for all their data center operations.

Re: New LLM optimization technique slashes memory costs

#78

Post by pizza »

richwater wrote: post #7

This is for inference right? Not training?

It's for KV caching. In most conversations that will mean inference. But you can do reinforcement learning using sampled sequences, and you could use KV caching to speed that up too, so that would be an instance where training could get a slight boost.

Re: New LLM optimization technique slashes memory costs

#79

Post by pizza »

h0l0cube wrote: post #41
hackernewds wrote: post #34

Earlier quoted context omitted.

Like what?

Oh, I don’t know, how about reducing the search space/accelerating the search speed for potential room temperature superconductors? Or how about the same for viable battery chemistries?

Would you like that with or without tokens?

Re: New LLM optimization technique slashes memory costs

#80

Post by baq »

owenpalmer wrote: post #67
odyssey7 wrote: post #36

Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?

No. This is a classic case of Jevon's paradox. Increased efficiency in resource use can lead to increased consumption of that resource, rather than decreased consumption. Example: 1. To decrease total gas consumption, more fuel efficient vehicles are invented. 2. Instead of using less gas, people drive more miles . They take longer road trips, commute farther for work, and more people can now afford to drive. 3. This…

See also https://en.m.wikipedia.org/wiki/Induced_demand
Post reply on HN