The RAM shortage could last years
theverge.com
The RAM shortage could last years
1–10 of 536 posts
Re: The RAM shortage could last years
#2can't read the article due to a paywall.
Re: The RAM shortage could last years
#3I'm a bit surprised the article makes no mention of Google's TurboQuant[0] introduced 26 days prior.
Given that TurboQuant results in a 6x reduction in memory usage for KV caches and up to 8x boost in speed, this optimization is already showing up in llama.cpp, enabling significantly bigger contexts without having to run a smaller model to fit it all in memory.
Some people thought it might significantly improve the RAM situation, though I remain a bit skeptical - the demand is probably still larger than the reduction turboquant brings.
Re: The RAM shortage could last years
#4can't read the article due to a paywall.
here is the source: https://www.reuters.com/world/asia-pacific/south-koreas-sk-g...
Re: The RAM shortage could last years
#5The era of optimisation is finally here. I'm excited.
Re: The RAM shortage could last years
#6can't read the article due to a paywall.
Re: The RAM shortage could last years
#7I'm a bit surprised the article makes no mention of Google's TurboQuant[0] introduced 26 days prior. Given that TurboQuant results in a 6x reduction in memory usage for KV caches and up to 8x boost in speed, this optimization is already showing up in llama.cpp, enabling significantly bigger contexts without having to run a smaller model to fit it all in memory. Some people thought it might significantly improve the R…
You can still use as much memory, but fit more things into it, so I don’t think the current market hogs will let go easily.
Re: The RAM shortage could last years
#8I'm a bit surprised the article makes no mention of Google's TurboQuant[0] introduced 26 days prior. Given that TurboQuant results in a 6x reduction in memory usage for KV caches and up to 8x boost in speed, this optimization is already showing up in llama.cpp, enabling significantly bigger contexts without having to run a smaller model to fit it all in memory. Some people thought it might significantly improve the R…
You still need to hold the model in memory. If you have for example 16 GB ram, the gains aren't that much
Re: The RAM shortage could last years
#9The era of optimisation is finally here. I'm excited.
Wait until China invades Taiwan.. (ok, it's not too likely, but what if?)
Re: The RAM shortage could last years
#10I fear that the real reason we do have a shortage, I mean, the real reason for the demand, is AI companies scooping what they can so that their competitors, whether existing or incumbent, can’t get to it.