New LLM optimization technique slashes memory costs • Hacker News Board

Re: New LLM optimization technique slashes memory costs

#181

Post by _aavaa_ » Tue, Dec 17, 2024, 7:26 PM UTC

yunohn wrote: post #172

_aavaa_ wrote: post #137

Earlier quoted context omitted.

My tone is because this is a simple predatory delay strategy. Tomorrow, tomorrow, I’ll decarbonize tomorrow. Instead of paying to buy wind and solar plants, which can go up today they are signing a meaningless agreement for the future . A PPA isn’t worth the paper it’s written on if the seller can’t produce electricity at the agreed upon price by the date required. Take Three Mile Island. It was closed in 2019 since…

> Instead of paying to buy wind and solar plants Have you considered googling and checking your assumptions? May help clear up the cynical misunderstandings you appear to have. If you had, you would’ve read that both Microsoft and Google invest heavily into wind and solar, and that Google is the largest corporate purchaser of renewables in the world. I’m not advocating for these companies, just trying to show that te…

> Have you considered googling and checking your assumptions? May help clear up the cynical misunderstandings you appear to have.

I don't have any such misunderstanding. Perhaps consider seeing my original comment which links to an article describing Google building out solar and wind farms for its data centres.

My cynicism, which I argue is well founded, is based around tech companies signing such agreements with nuclear companies, especially when it involves doings things that have never been done before (restarting reactors and building economical SMRs, see Nuscale...).

All these agreements are likely to amount to nothing more than positive PR, greenwashing, or predatory delay. Yes, they also build out solar and wind, but their nuclear PPAs are given equal standing with projects which actually are likely to be built; so instead of having to build more solar and wind today for more real money, they can promise to buy nuclear tomorrow for no cost today.

Re: New LLM optimization technique slashes memory costs

#182

Post by xpl » Tue, Dec 17, 2024, 7:28 PM UTC

cs702 wrote: post #161

Very clever, very meta, and it seems to work really well. The two big take-aways for me are: * It's possible to train a model to learn to summarize context from the attention matrix , based only on dot-product scores (k @ q.T * mask), regardless of how tokens are embedded . * Once the model is trained, it will work with any attention matrix, even if it's the attention matrix of another model. I've added this to my ev…

Is there any intuition why does it even work? It seems very unexpected.

Re: New LLM optimization technique slashes memory costs

#183

Post by momojo » Tue, Dec 17, 2024, 7:44 PM UTC

mandmandam wrote: post #123

solarkraft wrote: post #54

It’s mind bogglingly crazy that language models rivaling ones that used to require huge GPUs with a ton of VRAM to run now run on my upper-mid-range laptop from 4 years ago. At usable speed. Crazy. I didn’t expect capable language models to be practical/possible to run loyally, much less on hardware I already have.

> I didn’t expect capable language models to be practical/possible to run loyally Now there's a fun typo. Hopefully not too much fun.

That sentence triggered memories of reading the huge paperback Asimov compilation my mom kept on the bookshelf as a kid.

Re: New LLM optimization technique slashes memory costs

#184

Post by mouse_ » Tue, Dec 17, 2024, 7:50 PM UTC

bongodongobob wrote: post #63

asdev wrote: post #28

Earlier quoted context omitted.

no one is successfully using LLMs for anything other than customer service related things and text generation(coding, writing)

Rubbish. I built a pipeline to handle document classification that successfully took care of ~70TB of mostly unstructured and unorganized data, by myself, in a couple weeks, with no data engineering background whatsoever. This was quite literally impossible a couple years ago. The amount of work that saved was massive and is going to save us a shit ton of money on storage costs. Decades worth of invoices and random P…

That's fantastic. Congratulations!

Re: New LLM optimization technique slashes memory costs

#185

Post by [deleted] » Tue, Dec 17, 2024, 7:52 PM UTC

baq wrote: post #82

solarkraft wrote: post #54

It’s mind bogglingly crazy that language models rivaling ones that used to require huge GPUs with a ton of VRAM to run now run on my upper-mid-range laptop from 4 years ago. At usable speed. Crazy. I didn’t expect capable language models to be practical/possible to run loyally, much less on hardware I already have.

You have a sota multi-modal LLM running in your head at 20W, shared with best in class sensor package and top performing robotics control unit. There’s soooo much more to optimize.

[deleted]

Re: New LLM optimization technique slashes memory costs

#186

Post by exe34 » Tue, Dec 17, 2024, 7:54 PM UTC

ViewTrick1002 wrote: post #180

Eisenstein wrote: post #166

Earlier quoted context omitted.

> American companies and utilities announced 30 reactors. Britain announced ~14. Lots of projects get announced, they aren't meant to be promises. > The end result of what we broke ground on is 3 cancelled reactors, 3 reactors which entered commercial operation in the 2020s and 1 still under construction. So there are three operational reactors and another one almost ready. I'm surprised we got that after Fukushima.…

Which would have moved forward towards completion if the economic calculus made sense. We should of course continue with basic research. But, without some incredible breakthrough nuclear power will only serve climate change deniers agenda in delaying the renewable buildout. This is what you sign up for when proposing investing in nuclear power in 2024: > The opposition last week released modelling of its “coal-to-nuc…

in other words, it's too late to build nuclear, let's bury our heads in the sand and hope somehow we have enough renewable in 20 years and we're not still using the coal/gas.

Re: New LLM optimization technique slashes memory costs

#187

Post by ViewTrick1002 » Tue, Dec 17, 2024, 8:28 PM UTC

exe34 wrote: post #186

ViewTrick1002 wrote: post #180

Earlier quoted context omitted.

Which would have moved forward towards completion if the economic calculus made sense. We should of course continue with basic research. But, without some incredible breakthrough nuclear power will only serve climate change deniers agenda in delaying the renewable buildout. This is what you sign up for when proposing investing in nuclear power in 2024: > The opposition last week released modelling of its “coal-to-nuc…

in other words, it's too late to build nuclear, let's bury our heads in the sand and hope somehow we have enough renewable in 20 years and we're not still using the coal/gas.

The bury our heads in the sand part seems to be you projecting.

The research disagrees with you. Whenever new built nuclear power is included in the analysis the results becomes prohibitively expensive.

> Focusing on the case of Denmark, this article investigates a future fully sector-coupled energy system in a carbon-neutral society and compares the operation and costs of renewables and nuclear-based energy systems.

> The study finds that investments in flexibility in the electricity supply are needed in both systems due to the constant production pattern of nuclear and the variability of renewable energy sources.

> However, the scenario with high nuclear implementation is 1.2 billion EUR more expensive annually compared to a scenario only based on renewables, *with all systems completely balancing supply and demand across all energy sectors in every hour*.

> For nuclear power to be cost competitive with renewables an investment cost of 1.55 MEUR/MW must be achieved, which is substantially below any cost projection for nuclear power.

https://www.sciencedirect.com/science/article/pii/S030626192...

Or if you want a more southern latitude you have Australia here:

https://www.csiro.au/-/media/Energy/GenCost/GenCost2024-25Co...

Re: New LLM optimization technique slashes memory costs

#188

Post by [deleted] » Tue, Dec 17, 2024, 8:44 PM UTC

ViewTrick1002 wrote: post #187

exe34 wrote: post #186

Earlier quoted context omitted.

in other words, it's too late to build nuclear, let's bury our heads in the sand and hope somehow we have enough renewable in 20 years and we're not still using the coal/gas.

The bury our heads in the sand part seems to be you projecting. The research disagrees with you. Whenever new built nuclear power is included in the analysis the results becomes prohibitively expensive. > Focusing on the case of Denmark, this article investigates a future fully sector-coupled energy system in a carbon-neutral society and compares the operation and costs of renewables and nuclear-based energy systems.…

[deleted]

Re: New LLM optimization technique slashes memory costs

#189

Post by Eisenstein » Tue, Dec 17, 2024, 8:56 PM UTC

ViewTrick1002 wrote: post #187

exe34 wrote: post #186

Earlier quoted context omitted.

in other words, it's too late to build nuclear, let's bury our heads in the sand and hope somehow we have enough renewable in 20 years and we're not still using the coal/gas.

The bury our heads in the sand part seems to be you projecting. The research disagrees with you. Whenever new built nuclear power is included in the analysis the results becomes prohibitively expensive. > Focusing on the case of Denmark, this article investigates a future fully sector-coupled energy system in a carbon-neutral society and compares the operation and costs of renewables and nuclear-based energy systems.…

It may cost more, but it is constant generation, and we should invest in as many carbon neutral alternatives as possible that are feasible. The fact that you have a political opposition to it because of conservative opportunists using it for their own agenda is irrelevant.

Re: New LLM optimization technique slashes memory costs

#190

Post by ViewTrick1002 » Tue, Dec 17, 2024, 9:13 PM UTC

Eisenstein wrote: post #189

ViewTrick1002 wrote: post #187

Earlier quoted context omitted.

The bury our heads in the sand part seems to be you projecting. The research disagrees with you. Whenever new built nuclear power is included in the analysis the results becomes prohibitively expensive. > Focusing on the case of Denmark, this article investigates a future fully sector-coupled energy system in a carbon-neutral society and compares the operation and costs of renewables and nuclear-based energy systems.…

It may cost more, but it is constant generation, and we should invest in as many carbon neutral alternatives as possible that are feasible. The fact that you have a political opposition to it because of conservative opportunists using it for their own agenda is irrelevant.

Which is not what any modern grid needs? We need cheap dispatchable power, not horrifically expensive inflexible power.

Many grids around the world already spend loads of time with renewables filling 100% of the demand.

https://www.power-technology.com/news/california-achieves-10...

That is a down right hostile environment for nuclear power which relies on being able to output at 100% 24/7 all year around to only be horrifically expensive.

In the land of infinite resources and infinite time "all of the above" is a viable answer. In the real world we neither have infinite resources nor infinite time to fix climate change.

Lets focus our limited resources on what works and instead spend the big bucks on decarbonizing truly hard areas like aviation, construction, shipping and agriculture.