Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New LLM optimization technique slashes memory costs

venturebeat.com

Post reply on HN
Print view View original

171–180 of 227 posts

Re: New LLM optimization technique slashes memory costs

#171

Post by vlovich123 »

oblio wrote: post #130
tsukikage wrote: post #114

Earlier quoted context omitted.

This isn't like lossless compression. Both techniques involve throwing lots of information away, with the justification that doing so does not significantly affect the end result. The extent to which using both the techniques together will help will depend on how much overlap there is between the information each ends up discarding.

My joke was more along the lines of entropy. Entropy is information and you can't throw away all of it, otherwise you have nothing useful left.

Modern LLMs are still quite inefficient in their representation of information. We're at like the DEFLATE era and we've still yet to invent zstd where there's only marginal incremental gains; so right now there's a lot of waste to prune away.

Re: New LLM optimization technique slashes memory costs

#172

Post by yunohn »

_aavaa_ wrote: post #137
yunohn wrote: post #76

Earlier quoted context omitted.

Look, I agree that nuclear is difficult, but Google and Microsoft have publicly committed to those projects you’re mentioning. I don’t understand your dismissive tone that all of it is hogwash? This is one of those HN armchair comments.

My tone is because this is a simple predatory delay strategy. Tomorrow, tomorrow, I’ll decarbonize tomorrow. Instead of paying to buy wind and solar plants, which can go up today they are signing a meaningless agreement for the future . A PPA isn’t worth the paper it’s written on if the seller can’t produce electricity at the agreed upon price by the date required. Take Three Mile Island. It was closed in 2019 since…

> Instead of paying to buy wind and solar plants

Have you considered googling and checking your assumptions? May help clear up the cynical misunderstandings you appear to have.

If you had, you would’ve read that both Microsoft and Google invest heavily into wind and solar, and that Google is the largest corporate purchaser of renewables in the world. I’m not advocating for these companies, just trying to show that tech is one of the few industries that does actually care and invest into clean energy.

Some sources:

- https://www.gstatic.com/gumdrop/sustainability/google-2024-e...

- https://www.latitudemedia.com/news/googles-largest-wind-inve...

- https://amp.theguardian.com/technology/2019/sep/20/google-sa...

- https://www.theverge.com/2024/5/2/24147153/microsoft-ai-data...

Re: New LLM optimization technique slashes memory costs

#173

Post by adelpozo »

cratermoon wrote: post #167
adelpozo wrote: post #73

Earlier quoted context omitted.

Or solar in space, which some have already heard of Lumen Orbit https://www.ycombinator.com/companies/lumen-orbit

Space-based solar power contains little intrinsic advantage that we can get “only from space.” It looks like a wash at best, and the astronomers would say “don’t bother.” https://dothemath.ucsd.edu/2012/03/space-based-solar-power/

Yeah, but what I found thought provoking is what if you send the solar panels and the datacenter as well for training. No need to transmission of power down to earth. I guess then it becomes a heat dissipation and hardware upgrade and maintenance. But again, thought provoking.

Re: New LLM optimization technique slashes memory costs

#174

Post by crazygringo »

bonoboTP wrote: post #98
owenpalmer wrote: post #67

Earlier quoted context omitted.

No. This is a classic case of Jevon's paradox. Increased efficiency in resource use can lead to increased consumption of that resource, rather than decreased consumption. Example: 1. To decrease total gas consumption, more fuel efficient vehicles are invented. 2. Instead of using less gas, people drive more miles . They take longer road trips, commute farther for work, and more people can now afford to drive. 3. This…

There's no paradox in that. People became more capable and can afford to do more.

Really, once you understand any "paradox", it ceases to be a paradox at all.

I always feel a bit silly referring to any "paradox" as such, when it's not a paradox anymore. When it now makes perfect sense.

Re: New LLM optimization technique slashes memory costs

#175

Post by bonoboTP »

oreilles wrote: post #121
bonoboTP wrote: post #98

Earlier quoted context omitted.

There's no paradox in that. People became more capable and can afford to do more.

It is a paradox because there is an apparent contradiction in the fact that higher efficiency leads to higher consumption. By definition the opposite should be true.

I don't share that intuition. If I earn more money, I won't necessarily save more. I'll buy better food, better clothes, better everything and live a materially more prosperous life. My savings rate may even go down. Or up. It depends on the specifics. When a tech gets more efficient, it causes people to do more. To shape their surroundings and bend reality more to their will. If you can travel easier, you can realize your travel wishes better.

Re: New LLM optimization technique slashes memory costs

#177

Post by bonoboTP »

HPsquared wrote: post #142
michaelhoney wrote: post #110

Earlier quoted context omitted.

Yes, the term is a bit clumsy. The way I think of it, people have desires (to drive on the highway), but are dissuaded from doing so by disincentives (it’s too busy). Adding a lane reduces the disincentive, so that latent desire is satisfied, until it reaches a new equilibrium.

Result: more people getting where they want to go.

But sometimes the social environment adapts and now you have to drive that amount because it got factored in and things are now built further away, so whether you want to go far is not up to you. See long commutes becoming the norm. Now, arguably long commutes lead to better job allocations and more efficient land use as people can live in one place and work in any of the workplaces within a large radius. So it's a bit more complicated than "want", but ultimately more value seems to be produced.

Re: New LLM optimization technique slashes memory costs

#178

Post by benatkin »

vlovich123 wrote: post #168
benatkin wrote: post #164

Earlier quoted context omitted.

I thought the context would also have floating point numbers so that tokens would be included in a more fuzzy way, and that when requests are sent it would result in loading slightly different tokens into the cache. Yeah my understanding certainly is limited and I’d like to study it more. Thanks for the response, I see more similarity now.

The word you're looking for is latent space and yes, everything in the compute graph, including context cache & compute is done in latent space. Literal input tokens are first converted to latent space through the embedding layer and literal output tokens are generated by converting the last compute tensor into token probabilities & taking the most probable token. Everything in the middle though happens in the "float…

The KV cache feels more like a graph to me, like in the RDF sense. Each parameter could be numbered and given a URL it seems. I have some studying to do. I think building a simple neural net and looking at raw data for context in whatever LLM I’m playing with in Ollama are good things to try.

Re: New LLM optimization technique slashes memory costs

#179

Post by kaibee »

bonoboTP wrote: post #177
HPsquared wrote: post #142

Earlier quoted context omitted.

Result: more people getting where they want to go.

But sometimes the social environment adapts and now you have to drive that amount because it got factored in and things are now built further away, so whether you want to go far is not up to you. See long commutes becoming the norm. Now, arguably long commutes lead to better job allocations and more efficient land use as people can live in one place and work in any of the workplaces within a large radius. So it's a b…

> ultimately more value seems to be produced.

Is more value produced or are costs just shifted off the balance sheet onto the public commons? Driving instead of walking/public transit has certainly been profitable for some people/companies. But it has also been less than ideal from a public health standpoint. And the time spent commuting is unpaid, so while the business saves money on rent, the increase in travel time is still a cost borne by society as a whole. I would describe this as the opposite of 'efficient land use' personally.

Re: New LLM optimization technique slashes memory costs

#180

Post by ViewTrick1002 »

Eisenstein wrote: post #166
ViewTrick1002 wrote: post #149

Earlier quoted context omitted.

There was talk of an ongoing nuclear renaissance in the early 2000s. [1] American companies and utilities announced 30 reactors. Britain announced ~14. We went ahead and started construction on 7 reactors in Vogtle, Virgil C. Summer, Flamanville, Olkiluoto and Hanhikivi to rekindle the industry. We didn't believe renewables would cut it. The end result of what we broke ground on is 3 cancelled reactors, 3 reactors wh…

> American companies and utilities announced 30 reactors. Britain announced ~14. Lots of projects get announced, they aren't meant to be promises. > The end result of what we broke ground on is 3 cancelled reactors, 3 reactors which entered commercial operation in the 2020s and 1 still under construction. So there are three operational reactors and another one almost ready. I'm surprised we got that after Fukushima.…

Which would have moved forward towards completion if the economic calculus made sense.

We should of course continue with basic research. But, without some incredible breakthrough nuclear power will only serve climate change deniers agenda in delaying the renewable buildout.

This is what you sign up for when proposing investing in nuclear power in 2024:

> The opposition last week released modelling of its “coal-to-nuclear” plan that would slow the rollout of renewable energy and batteries and instead rely on more fossil fuel generation until a nuclear industry could be developed, mostly after 2040.

https://www.theguardian.com/australia-news/2024/dec/16/coali...

Post reply on HN