Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New LLM optimization technique slashes memory costs

venturebeat.com

Post reply on HN
Print view View original

191–200 of 227 posts

Re: New LLM optimization technique slashes memory costs

#191

Post by Eisenstein »

ViewTrick1002 wrote: post #190
Eisenstein wrote: post #189

Earlier quoted context omitted.

It may cost more, but it is constant generation, and we should invest in as many carbon neutral alternatives as possible that are feasible. The fact that you have a political opposition to it because of conservative opportunists using it for their own agenda is irrelevant.

Which is not what any modern grid needs? We need cheap dispatchable power, not horrifically expensive inflexible power. Many grids around the world already spend loads of time with renewables filling 100% of the demand. https://www.power-technology.com/news/california-achieves-10... That is a down right hostile environment for nuclear power which relies on being able to output at 100% 24/7 all year around to only be…

'Plenty of places' is not all places and you want to completely count out a significant energy generating ability because you are annoyed that it doesn't agree with your politics. If it isn't feasible then they won't build it -- by going around and advocating against it you are doing the same thing that happened in the 70s and 80s -- removing a perfectly valid option for energy that we need and will otherwise be fulfilled in any other way if not provided -- almost always with fossil fuels. If you can guarantee every place for all time will be fine with renewables, I'd like to see it, otherwise, why not step back and let engineers and scientists evaluate instead of grandstanding against an option?

Re: New LLM optimization technique slashes memory costs

#192

Post by ViewTrick1002 »

Eisenstein wrote: post #191
ViewTrick1002 wrote: post #190

Earlier quoted context omitted.

Which is not what any modern grid needs? We need cheap dispatchable power, not horrifically expensive inflexible power. Many grids around the world already spend loads of time with renewables filling 100% of the demand. https://www.power-technology.com/news/california-achieves-10... That is a down right hostile environment for nuclear power which relies on being able to output at 100% 24/7 all year around to only be…

'Plenty of places' is not all places and you want to completely count out a significant energy generating ability because you are annoyed that it doesn't agree with your politics. If it isn't feasible then they won't build it -- by going around and advocating against it you are doing the same thing that happened in the 70s and 80s -- removing a perfectly valid option for energy that we need and will otherwise be fulf…

What places aren’t covered by the spectrum with Denmark for higher latitudes and Australia for the near the equator?

I’m advocating against wasting public money on nuclear power pretending it is a solution to climate change.

Have at it with your own money.

I already provided you with the scientists and engineers, but you seem to have completely disregarded them because they did not align with what you wanted.

I can do it again:

The research disagrees with you. Whenever new built nuclear power is included in the analysis the results becomes prohibitively expensive.

> Focusing on the case of Denmark, this article investigates a future fully sector-coupled energy system in a carbon-neutral society and compares the operation and costs of renewables and nuclear-based energy systems.

> The study finds that investments in flexibility in the electricity supply are needed in both systems due to the constant production pattern of nuclear and the variability of renewable energy sources.

> However, the scenario with high nuclear implementation is 1.2 billion EUR more expensive annually compared to a scenario only based on renewables, *with all systems completely balancing supply and demand across all energy sectors in every hour*.

> For nuclear power to be cost competitive with renewables an investment cost of 1.55 MEUR/MW must be achieved, which is substantially below any cost projection for nuclear power.

https://www.sciencedirect.com/science/article/pii/S030626192...

Or if you want a more southern latitude you have Australia here:

https://www.csiro.au/-/media/Energy/GenCost/GenCost2024-25Co...

Re: New LLM optimization technique slashes memory costs

#193

Post by cs702 »

xpl wrote: post #182
cs702 wrote: post #161

Very clever, very meta, and it seems to work really well. The two big take-aways for me are: * It's possible to train a model to learn to summarize context from the attention matrix , based only on dot-product scores (k @ q.T * mask), regardless of how tokens are embedded . * Once the model is trained, it will work with any attention matrix, even if it's the attention matrix of another model. I've added this to my ev…

Is there any intuition why does it even work? It seems very unexpected.

The intuition is that the relative frequency at which past tokens get attention from future tokens is a good proxy for their relative importance.

The model the authors use, in fact, maps attention scores to features in the frequency domain.

Re: New LLM optimization technique slashes memory costs

#194

Post by gavmor »

Eisenstein wrote: post #170
prox wrote: post #88

Earlier quoted context omitted.

I might have a go at installing one, what is a good source or install at the moment?

If you don't care about docker packages being used as installers and your home directory invisibly used to store massive weight files in exchange for not having to deal with learning any configuration: ollama or lmstudio. If you just want to play for a bit: llamafile If you want granular control with ease of execution in exchange for having to figure out what the settings mean and figure out which weights to download…

I run ollama off a symlink to an external volume. It just feels neater that way, and can run any GGUF off of HuggingFace. I would like to know what configuration I'm missing out on, though.

Re: New LLM optimization technique slashes memory costs

#195

Post by abrichr »

punkpeye wrote: post #153
vlovich123 wrote: post #5

Wonder how this compares with Microsoft's HeadKV paper [1] which claims a 98% percent reduction in memory while retaining 97% of the performance. [1] https://arxiv.org/html/2410.19258v3

Any real-world (open-source) implementations of this?

Looks like it is open source: https://github.com/FYYFU/HeadKV

Re: New LLM optimization technique slashes memory costs

#196

Post by K0balt »

eru wrote: post #49
K0balt wrote: post #25

Earlier quoted context omitted.

Finance is basically all of the reasons not to use (generative, LLM based) AI , all in one vertical. The poster child of determinism.

Could you please explain? Finance is a big industry, and they are doing lots of different things.

Sure, there’s lots of room for LLMs in helping to do clerical work, HR, that kind of thing. I was actually thinking of the direct management of funds and investments. So yeah, like probably all businesses, the ancillary functions can probably improve productivity using Generative AI with a minimal hit to quality.

Re: New LLM optimization technique slashes memory costs

#197

Post by bongodongobob »

mewpmewp2 wrote: post #129
nwmcsween wrote: post #95

Earlier quoted context omitted.

I mean considering I did document classification back in 2010 using tesseract, I wouldn't say it was impossible.

But obviously it would be far from accuracy that LLM would be able to do. E.g. generate search keywords, tags, other type of meta data for a certain document.

Yup that's exactly it. By being able to tag things with all sorts of in house meta data we were then able to search and group things extremely accurately. There was still a lot of human in the mix, but this made the whole task going from "idk if we can even consider doing this" to "great, we can break this down and chip away at it over the next few months/throw some interns at it".

Re: New LLM optimization technique slashes memory costs

#198

Post by bongodongobob »

aitchnyu wrote: post #71
bongodongobob wrote: post #63

Earlier quoted context omitted.

Rubbish. I built a pipeline to handle document classification that successfully took care of ~70TB of mostly unstructured and unorganized data, by myself, in a couple weeks, with no data engineering background whatsoever. This was quite literally impossible a couple years ago. The amount of work that saved was massive and is going to save us a shit ton of money on storage costs. Decades worth of invoices and random P…

Could you describe your stack and how its much more effective than two years ago? I heard of printed-table OCR and doc classification years back.

Tagging with in house metadata like division, job code, who was the project manager etc.

Re: New LLM optimization technique slashes memory costs

#199

Post by bongodongobob »

musha68k wrote: post #74
bongodongobob wrote: post #63

Earlier quoted context omitted.

Rubbish. I built a pipeline to handle document classification that successfully took care of ~70TB of mostly unstructured and unorganized data, by myself, in a couple weeks, with no data engineering background whatsoever. This was quite literally impossible a couple years ago. The amount of work that saved was massive and is going to save us a shit ton of money on storage costs. Decades worth of invoices and random P…

Very interesting. If I may ask: how are you handling the correctness issue? What's the workflow there if even able to spot a mishap?

We came up with different categories of tags. I should clarify, the AI didn't actually do the sorting, it did tagging so sorting was tractable. After the tagging it's just a matter of grouping, either by algorithm or human.

Re: New LLM optimization technique slashes memory costs

#200

Post by yunohn »

_aavaa_ wrote: post #181
yunohn wrote: post #172

Earlier quoted context omitted.

> Instead of paying to buy wind and solar plants Have you considered googling and checking your assumptions? May help clear up the cynical misunderstandings you appear to have. If you had, you would’ve read that both Microsoft and Google invest heavily into wind and solar, and that Google is the largest corporate purchaser of renewables in the world. I’m not advocating for these companies, just trying to show that te…

> Have you considered googling and checking your assumptions? May help clear up the cynical misunderstandings you appear to have. I don't have any such misunderstanding. Perhaps consider seeing my original comment which links to an article describing Google building out solar and wind farms for its data centres. My cynicism, which I argue is well founded, is based around tech companies signing such agreements with nu…

Not one of your comments amidst this sprawling thread has a single positive fact in it. You’re blindly arguing “nuclear bad” and claiming that nuclear PPAs amount to nothing based on zero evidence? The ones we’re all discussing are the first of their kind…

Anyways, it’s been a bore, cheers!

Post reply on HN