151–160 of 227 posts
Earlier quoted context omitted.
I might have a go at installing one, what is a good source or install at the moment?
Ollama was the easiest way to set up local LLMs for me. https://ollama.com/
Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?
No. This is a classic case of Jevon's paradox. Increased efficiency in resource use can lead to increased consumption of that resource, rather than decreased consumption. Example: 1. To decrease total gas consumption, more fuel efficient vehicles are invented. 2. Instead of using less gas, people drive more miles . They take longer road trips, commute farther for work, and more people can now afford to drive. 3. This…
Wonder how this compares with Microsoft's HeadKV paper [1] which claims a 98% percent reduction in memory while retaining 97% of the performance. [1] https://arxiv.org/html/2410.19258v3
Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?
The grave implication of Jevons paradox is that the fundamental conflict between sustainability and economic progress is not resolved solely by using resources more efficiently. It's a theory of supply chain constraints essentially. Once a resource is used more efficiently, its use is increased until the next most economically constrained resource hits its economically useful limit.
Earlier quoted context omitted.
Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Others are trying to restart an existing reactor at three mile island (a thing that has never been done before, and likely won't be done now since the reactor was shu…
And unfortunately, gas and coal in the meantime. https://www.theguardian.com/technology/2024/sep/15/data-cent...
Electric cars are causing exactly the same problem.
Also the "recs" appear to be based on a lie. Overall, an increase in load can only be green if there is added new green generation to service that load.
Earlier quoted context omitted.
Yes, the term is a bit clumsy. The way I think of it, people have desires (to drive on the highway), but are dissuaded from doing so by disincentives (it’s too busy). Adding a lane reduces the disincentive, so that latent desire is satisfied, until it reaches a new equilibrium.
Result: more people getting where they want to go.
Earlier quoted context omitted.
You have a sota multi-modal LLM running in your head at 20W, shared with best in class sensor package and top performing robotics control unit. There’s soooo much more to optimize.
I would argue that our sensor package is losing its lead very quickly-- audio performance is already on par with current tech and image processing is closing the gap very quickly as well (it helps a lot that silicon-based technology is much less constrained on bandwidth). Tactile sensing is still lightyears ahead, and I don't see that situation improving anytime soon...
Earlier quoted context omitted.
They’re both exploring the same space of optimizing the memory needed by the KV cache which is essentially another name for the context window (no one elides the KV cache as otherwise you’re doing N^2 math to do attention). They’re exploring different approaches to achieve the same goal and they may be both possible to apply simultaneously to reduce the attention mechanism to almost 0 memory usage which would be real…
That sounds like a stretch to me. If not, I’m impressed how the articles can describe such similar things in such different terms.
The articles use very similar verbiage.
> The context window can be considered the model’s working memory
Snip
> Universal transformer memory optimizes prompts using neural attention memory models (NAMMs), simple neural networks that decide whether to “remember” or “forget” each given token stored in the LLM’s memory.
snip
> Meanwhile, by discarding unnecessary tokens, NAMM enabled the LLM model to save up to 75% of its cache memory while performing the tasks.
You just have to be familiar with the wording in the space and read enough literature. Here’s more direct wording from the NAMM paper:
> NAMMs use evolution to optimize the performance of LMs by pruning their KV cache memory. Evolved NAMMs can be zero-shot transferred to other transformers, even across input modalities and task domains.
This is all related work about shrinking the size of the KV cache as the context grows both due to memory and it also has a speed up effect since you’re not having to attend all the tokens (O(n) -> sublinear with the size of the context).
Context is critical in the LLM answering correctly and remembering all the information given to it + everything it said. Typical limits for open models these days are 128k but with techniques like this it could scale even further allowing better performance on thing like code completion.
Earlier quoted context omitted.
And unfortunately, gas and coal in the meantime. https://www.theguardian.com/technology/2024/sep/15/data-cent...
That article is terribly vague. Electric cars are causing exactly the same problem. Also the "recs" appear to be based on a lie. Overall, an increase in load can only be green if there is added new green generation to service that load.
The alternatives for these data centres are either build renewables or not build the data centres, both of which are better.
Is it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?