Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What if AI doesn't need more RAM but better math?

adlrocha.substack.com

Post reply on HN
Print view View original

61–70 of 111 posts

Re: What if AI doesn't need more RAM but better math?

#61

Post by throwatdem12311 »

naasking wrote: post #35
throwatdem12311 wrote: post #13

Earlier quoted context omitted.

The hyperscalers do not want us running models at the edge and they will spend infinite amounts of circular fake money to ensure hardware remains prohibitively expensive forever.

> and they will spend infinite amounts of circular fake money to ensure hardware remains prohibitively expensive forever. That's ridiculous, "infinite money" isn't a thing. They will spend as much as they can not because they want to keep local solutions out, but because it enables them to provide cheaper services and capture more of the market. We all eventually benefit from that.

Cheaper for who? For them maybe but certainly not for you or me.

Re: What if AI doesn't need more RAM but better math?

#62

Post by lucasfin000 »

ssyhape wrote: post #11
jurgenburgen wrote: post #4

Earlier quoted context omitted.

If models become more efficient we will move more of the work to local devices instead of using SaaS models. We’re still in the mainframe era of LLM.

[flagged]

MoE feels a lot more like engineering to me. You're routing around the problem rather than actually solving it. The real math gains are things like quantization schemes that change how information is actually represented. Whether that distinction matters long term probably will depend on whether we hit a capability wall first or an efficiency ceiling first.

Re: What if AI doesn't need more RAM but better math?

#63

Post by rainsford »

jurgenburgen wrote: post #4
LoganDark wrote: post #3

We will not see memory demand decrease because this will simply allow AI companies to run more instances. They still want an infinite amount of memory at the moment, no matter how AI improves.

If models become more efficient we will move more of the work to local devices instead of using SaaS models. We’re still in the mainframe era of LLM.

We moved from the mainframe era to desktops and smaller servers because computers got fast enough to do what we needed them to do locally. Centralized computing resources are still vastly more powerful than what's under your desk or in a laptop, but it doesn't matter because people generally don't need that much power for their daily tasks.

The problem with AI is that it's not obvious what the upper limit of capability demand might be. And until or if we get there, there will always be demand for the more capable models that run on centralized computing resources. Even if at some point I'm able to run a model on my local desktop that's equivalent to current Claude Opus, if what Anthropic is offering as a service is significantly better in a way that matters to my use case, I will still want to use the SaaS one.

Re: What if AI doesn't need more RAM but better math?

#64

Post by delecti »

dd8601fn wrote: post #58
delecti wrote: post #44

Earlier quoted context omitted.

As I understand this advancement, this doesn't let you run bigger models, it lets you maintain more chat context. So Anthropic and OpenAI won't need as much hardware running inference to serve their users, but it doesn't do much to make bigger models work on smaller hardware. Though I'm not an expert, maybe my understanding of the memory allocation is wrong.

Seems to me if the model and the kv cache are competing for the same pool of memory, then massively compressing the cache necessarily means more ram available for (if it fits) a larger model, no?

Yes, but the context is a comparatively smaller part of how much memory is used when running it locally for a single user, vs when running it on a server for public... serving.

Re: What if AI doesn't need more RAM but better math?

#66

Post by lelanthran »

rainsford wrote: post #63
jurgenburgen wrote: post #4

Earlier quoted context omitted.

If models become more efficient we will move more of the work to local devices instead of using SaaS models. We’re still in the mainframe era of LLM.

We moved from the mainframe era to desktops and smaller servers because computers got fast enough to do what we needed them to do locally. Centralized computing resources are still vastly more powerful than what's under your desk or in a laptop, but it doesn't matter because people generally don't need that much power for their daily tasks. The problem with AI is that it's not obvious what the upper limit of capabili…

> Even if at some point I'm able to run a model on my local desktop that's equivalent to current Claude Opus, if what Anthropic is offering as a service is significantly better in a way that matters to my use case, I will still want to use the SaaS one.

Only if it's competitively priced. You wouldn't want to use the SaaS if the breakeven in investment on local instances is a matter of months.

Right now people are shelling out for Claude Code and similar because for $200/m they can consume $10k/m of tokens. If you were actually paying $10k/m, than it makes sense to splurge $20k-$30k for a local instance.

Re: What if AI doesn't need more RAM but better math?

#68

Post by lelanthran »

throwatdem12311 wrote: post #24
topspin wrote: post #21

Earlier quoted context omitted.

> they will spend infinite amounts of circular fake money > forever If that's the plan (there is no plan) then it expires at some point, because it's a spiral and such spirals always bottom out.

And when that happens people STILL won’t be able to afford the hardware.

> And when that happens people STILL won’t be able to afford the hardware.

Of course they will - if that happens all these AI token providers won't have a use for all that hardware they bought. You'll be buying used H100s and H200s off eBay for pennies on the dollar.

Re: What if AI doesn't need more RAM but better math?

#69

Post by zozbot234 »

barbegal wrote: post #42

Does the KV cache really grow to use more memory than the model weights? The reduction in overall RAM relies on the KV cache being a substantial proportion of the memory usage but with very large models I can't see how that holds true.

For long context, yes this is at least plausible. And the latest models are reaching context lengths of 1M tokens or perhaps more.

Re: What if AI doesn't need more RAM but better math?

#70

Post by lelanthran »

naasking wrote: post #35
throwatdem12311 wrote: post #13

Earlier quoted context omitted.

The hyperscalers do not want us running models at the edge and they will spend infinite amounts of circular fake money to ensure hardware remains prohibitively expensive forever.

> and they will spend infinite amounts of circular fake money to ensure hardware remains prohibitively expensive forever. That's ridiculous, "infinite money" isn't a thing. They will spend as much as they can not because they want to keep local solutions out, but because it enables them to provide cheaper services and capture more of the market. We all eventually benefit from that.

> That's ridiculous, "infinite money" isn't a thing.

My reading of GP is that he was being sarcastic - "infinite amounts of circular fake money" is probably a reference to these circular deals going on.

If A hands B investment of $100, then B hands A $100 for purchase of hardware, A's equity in B, on paper, is $100, plus A has revenue of $100 (from B), which gives A total assets of $200.

Obviously it has to be shuffled more thoroughly, but that's the basic idea that I thought GP was referring to.

Post reply on HN