Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What if AI doesn't need more RAM but better math?

adlrocha.substack.com

Post reply on HN
Print view View original

71–80 of 111 posts

Re: What if AI doesn't need more RAM but better math?

#71

Post by zozbot234 »

rainsford wrote: post #63
jurgenburgen wrote: post #4

Earlier quoted context omitted.

If models become more efficient we will move more of the work to local devices instead of using SaaS models. We’re still in the mainframe era of LLM.

We moved from the mainframe era to desktops and smaller servers because computers got fast enough to do what we needed them to do locally. Centralized computing resources are still vastly more powerful than what's under your desk or in a laptop, but it doesn't matter because people generally don't need that much power for their daily tasks. The problem with AI is that it's not obvious what the upper limit of capabili…

The underlying advantage of local inference is that you're repurposing your existing hardware for free. You don't need your token spend to pay a share of the capex cost for datacenters that are large enough to draw gigawatts in power, you can just pay for your own energy use. Even though the raw energy cost per operation will probably be higher for local inference, the overall savings in hardware costs can still be quite real.

Re: What if AI doesn't need more RAM but better math?

#73

Post by Skunkleton »

The TurboQuant paper is from April 2025. I’m sure the major labs knew about it on, or even before, the day it published. Any impact it had would have been a year ago. Yet I keep seeing these posts and discuss completely ignoring this.

Can we please start talking about this in that context? We already know what TurboQuant will do to DRAM demand. We already know what it will do to context windows. There is no need to speculate. There is no need to panic sell stocks.

Re: What if AI doesn't need more RAM but better math?

#74

Post by throwatdem12311 »

lelanthran wrote: post #68
throwatdem12311 wrote: post #24

Earlier quoted context omitted.

And when that happens people STILL won’t be able to afford the hardware.

> And when that happens people STILL won’t be able to afford the hardware. Of course they will - if that happens all these AI token providers won't have a use for all that hardware they bought. You'll be buying used H100s and H200s off eBay for pennies on the dollar.

No they won’t they’re just going to get absorbed into Azure and AWS and used for generic GPU compute that you rent until they’re burned out trash.

Re: What if AI doesn't need more RAM but better math?

#76

Post by leptons »

mustyoshi wrote: post #34
jurgenburgen wrote: post #4

Earlier quoted context omitted.

If models become more efficient we will move more of the work to local devices instead of using SaaS models. We’re still in the mainframe era of LLM.

I don't see how we'll ever get to widespread local LLM. The power efficiency alone is a strong enough pressure to use centralized model providers. My 3090 running 24b or 32b models is fun, but I know I'm paying way more per token in electricity, on top of lower quality tokens. It's fun to run them locally, but for anything actually useful it's cheaper to just pay API prices currently.

AI is not cheap to run no matter where it is running. The price we get charged today for AI is a loss-leader. The actual cost is much higher, so much higher that the average paying user today would balk at what it actually costs to run. These AI companies are trying to get people hooked on their product, to get it integrated into every business and workflow that they can, then start raising prices.

Re: What if AI doesn't need more RAM but better math?

#77

Post by redanddead »

SphericalCowww wrote: post #59

I mean, since GPT-4, I believe the RAM is no longer creating the miracle that the LLM performance scales directly with the model size. At least ChatGPT itself convinced me that any decent-sized company can create a GPT4 equivalent in terms of model size, but limited by service options, like memory cache and hallucination handling. Companies buy RAM simply to ride the stock hype. I am no expert, so this is a shallow t…

share it on gh and make a show hn post about it, maybe you're right

the models are still very stupid atm something needs to change

Re: What if AI doesn't need more RAM but better math?

#79

Post by otabdeveloper4 »

miohtama wrote: post #51
konaraddi wrote: post #26

> applying this compression algorithm at scale may significantly relax the memory bottleneck issue. I don’t think they’re going to downsize though, I think the big players are just going to use the freed up memory for more workflows or larger models because the big players want to scale up. It’s a cat and mouse race for the best models.

It will also help with local inference, making AI without big players possible.

It's already possible. Post-training is vastly more important than model size. (There's bigtime diminishing returns with increasing model size.)
Post reply on HN