Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New LLM optimization technique slashes memory costs

venturebeat.com

Post reply on HN
Print view View original

121–130 of 227 posts

Re: New LLM optimization technique slashes memory costs

#121

Post by oreilles »

bonoboTP wrote: post #98
owenpalmer wrote: post #67

Earlier quoted context omitted.

No. This is a classic case of Jevon's paradox. Increased efficiency in resource use can lead to increased consumption of that resource, rather than decreased consumption. Example: 1. To decrease total gas consumption, more fuel efficient vehicles are invented. 2. Instead of using less gas, people drive more miles . They take longer road trips, commute farther for work, and more people can now afford to drive. 3. This…

There's no paradox in that. People became more capable and can afford to do more.

It is a paradox because there is an apparent contradiction in the fact that higher efficiency leads to higher consumption. By definition the opposite should be true.

Re: New LLM optimization technique slashes memory costs

#122

Post by lopis »

cma wrote: post #75
_aavaa_ wrote: post #48

Earlier quoted context omitted.

Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Others are trying to restart an existing reactor at three mile island (a thing that has never been done before, and likely won't be done now since the reactor was shu…

> Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Not building new, but I think Microsoft paying to restart a reactor at Three Mile Island for their datacenter is much more significant than you make the deals sound…

It's all been pretty much greenwashing to distract for the real impact of all the AI infrastructure in the energy and water supply.

Re: New LLM optimization technique slashes memory costs

#123

Post by mandmandam »

solarkraft wrote: post #54

It’s mind bogglingly crazy that language models rivaling ones that used to require huge GPUs with a ton of VRAM to run now run on my upper-mid-range laptop from 4 years ago. At usable speed. Crazy. I didn’t expect capable language models to be practical/possible to run loyally, much less on hardware I already have.

> I didn’t expect capable language models to be practical/possible to run loyally

Now there's a fun typo. Hopefully not too much fun.

Re: New LLM optimization technique slashes memory costs

#124

Post by lopis »

yunohn wrote: post #76
_aavaa_ wrote: post #48

Earlier quoted context omitted.

Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Others are trying to restart an existing reactor at three mile island (a thing that has never been done before, and likely won't be done now since the reactor was shu…

Look, I agree that nuclear is difficult, but Google and Microsoft have publicly committed to those projects you’re mentioning. I don’t understand your dismissive tone that all of it is hogwash? This is one of those HN armchair comments.

Because it's all marketing and greenwashing. They are training these models today using fossil fuels. By the time those nuclear reactors are online, they will have gobbled up literally every human creation to train their models multiple times and dried up several water sources.

Re: New LLM optimization technique slashes memory costs

#125

Post by [deleted] »

lopis wrote: post #124
yunohn wrote: post #76

Earlier quoted context omitted.

Look, I agree that nuclear is difficult, but Google and Microsoft have publicly committed to those projects you’re mentioning. I don’t understand your dismissive tone that all of it is hogwash? This is one of those HN armchair comments.

Because it's all marketing and greenwashing. They are training these models today using fossil fuels. By the time those nuclear reactors are online, they will have gobbled up literally every human creation to train their models multiple times and dried up several water sources.

[deleted]

Re: New LLM optimization technique slashes memory costs

#126

Post by mewpmewp2 »

popinman322 wrote: post #12
asdev wrote: post #8

[flagged]

Google Trends make it seem like we're out of the exponential growth phase for LLMs-- search interest is possibly plateauing. A decline in search interest outside of academia makes sense. The groups who can get by on APIs don't care so much how the sausage is made and just want to see prices come down. Interested parties have likely already found tools that work for them. There's definitely some academic interest outs…

But "Large language model" as a topic in google trends is still in its peak. Maybe just everyone who would be the audience is already knowledgeable about LLMs so why would Google Search trends be able to keep rising?

ChatGPT is at it's peak, and something like Claude is still rising.

Re: New LLM optimization technique slashes memory costs

#127

Post by mewpmewp2 »

aitchnyu wrote: post #71
bongodongobob wrote: post #63

Earlier quoted context omitted.

Rubbish. I built a pipeline to handle document classification that successfully took care of ~70TB of mostly unstructured and unorganized data, by myself, in a couple weeks, with no data engineering background whatsoever. This was quite literally impossible a couple years ago. The amount of work that saved was massive and is going to save us a shit ton of money on storage costs. Decades worth of invoices and random P…

Could you describe your stack and how its much more effective than two years ago? I heard of printed-table OCR and doc classification years back.

But LLM is obviously able to organize documents and data much more intelligently than any ML algorithm from the past.

Re: New LLM optimization technique slashes memory costs

#128

Post by mewpmewp2 »

musha68k wrote: post #74
bongodongobob wrote: post #63

Earlier quoted context omitted.

Rubbish. I built a pipeline to handle document classification that successfully took care of ~70TB of mostly unstructured and unorganized data, by myself, in a couple weeks, with no data engineering background whatsoever. This was quite literally impossible a couple years ago. The amount of work that saved was massive and is going to save us a shit ton of money on storage costs. Decades worth of invoices and random P…

Very interesting. If I may ask: how are you handling the correctness issue? What's the workflow there if even able to spot a mishap?

Organising data even if it's not 100% perfect is much better than completely unorganized data.

Re: New LLM optimization technique slashes memory costs

#129

Post by mewpmewp2 »

nwmcsween wrote: post #95
bongodongobob wrote: post #63

Earlier quoted context omitted.

Rubbish. I built a pipeline to handle document classification that successfully took care of ~70TB of mostly unstructured and unorganized data, by myself, in a couple weeks, with no data engineering background whatsoever. This was quite literally impossible a couple years ago. The amount of work that saved was massive and is going to save us a shit ton of money on storage costs. Decades worth of invoices and random P…

I mean considering I did document classification back in 2010 using tesseract, I wouldn't say it was impossible.

But obviously it would be far from accuracy that LLM would be able to do. E.g. generate search keywords, tags, other type of meta data for a certain document.

Re: New LLM optimization technique slashes memory costs

#130

Post by oblio »

tsukikage wrote: post #114
oblio wrote: post #100

Earlier quoted context omitted.

> they may be both possible to apply simultaneously to reduce the attention mechanism to almost 0 memory usage which would be really cool https://matt.might.net/articles/why-infinite-or-guaranteed-f...

This isn't like lossless compression. Both techniques involve throwing lots of information away, with the justification that doing so does not significantly affect the end result. The extent to which using both the techniques together will help will depend on how much overlap there is between the information each ends up discarding.

My joke was more along the lines of entropy. Entropy is information and you can't throw away all of it, otherwise you have nothing useful left.
Post reply on HN