Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory has grown to nearly two-thirds of AI chip component costs

epoch.ai

Post reply on HN
Print view View original

411–420 of 524 posts

Re: Memory has grown to nearly two-thirds of AI chip component costs

#411

Post by exceptione »

kristopolous wrote: post #390
byzantinegene wrote: post #336

Earlier quoted context omitted.

not if china gets into the picture

why not? i'm sure they can jump into the hustle. Increasing the availability doesn't mean decreasing the price ... people think those are intrinsically related - not so much. You can get a prada shirt for $2,000 ... as many as you'd like, for $2,000 a piece. No problem. They'll make the factories go burr all night long. Still $2,000.sweeping There's a bunch of things like this. $100 bills for instance ... a new entra…

  > why not? i'm sure they can jump into the hustle.
Not so quick. Critical difference is the relationship between enterprises and the state. In China, the state owns the enterprise, in one way or another. High costs of memory is a threat to the established Chinese electronics manufacturers. The Chinese state can optimize returns at a higher level than the one some petty chip manufacturer operates at, especially if doing so means it could gain coercive geopolitical strength, aka blackmailing.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#412

Post by zozbot234 »

EnPissant wrote: post #400
zozbot234 wrote: post #233

Earlier quoted context omitted.

MoE is fine. You can put the shared weights on the 5090 (will fit handily even for the largest models) and expert weights on CPU, possibly with weights offload from storage.

Even if you could fit a 500B model's expert weights in very fast system RAM, it would run so slow as to be useless.

That's really only "useless" if the only thing you care about is a quick real-time response. Contrary to common perception, MoE models do benefit from batching requests together even when run on a single node, you just have to ensure you have at least ~5 parallel requests in flight (and that's for the very sparsest models) to really see the aggregate benefit.

(Intuitively, that's because the issue of whether any active weights are being shared among requests - thus, any memory throughput is being reused - is a generalized birthday problem. That's why even having a few parallel requests is quite effective. Especially since the "random" choice of experts happens anew at any single layer, so there's a lot of independent samples.)

Re: Memory has grown to nearly two-thirds of AI chip component costs

#413

Post by baq »

brandensilva wrote: post #183
radialstub wrote: post #161

Earlier quoted context omitted.

The memory makers will not expand demand drastically. It is in the nature of their business to keep the market under-supplied, otherwise the following oversupply will kill them. Instead, supply is just rerouted from less profitable segments such as mobile and personal computing.

China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply. But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer. I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.

China can afford and has the political will and power to centrally plan parts of the economy it feels like planning. Cars are obvious examples and if dram is next, western manufacturers should brace for impact.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#414

Post by jmb99 »

dominotw wrote: post #282
brandensilva wrote: post #183

Earlier quoted context omitted.

China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply. But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer. I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.

man i keep thinking. why cant india get into stuff like this. Do their own manhattan project to build factories and tech for this and immigrate experts with high salaries.

I am typing this on my return flight home from a business trip in Delhi. There are many other areas the Indian government needs to be focussing on first.

I had a similar view to you ~2 weeks ago. Spending some time there very quickly made me realize that there’s a lot of other things that are much more pressing.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#415

Post by alexwwang »

noosphr wrote: post #371
alexwwang wrote: post #369

Earlier quoted context omitted.

It’s a horrible thought. Really horrible. You should come to China and work in those factories and mines for some years by yourself.

You should work in the Central African Republic coltan mines if you think anyone has a leg to stand on.

I am living in China and I think you cannot just talk with imagination.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#416

Post by alexwwang »

viccis wrote: post #394
alexwwang wrote: post #369

Earlier quoted context omitted.

It’s a horrible thought. Really horrible. You should come to China and work in those factories and mines for some years by yourself.

Why would I, as an American engineer and user of tech hardware from China for quite some time now, need to immerse myself in the Chinese factories as if they are somehow worse than other ones throughout the world? Thanks, please give my regards to Kash Patel.

You just cannot see your privileges.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#417

Post by EnPissant »

zozbot234 wrote: post #412
EnPissant wrote: post #400

Earlier quoted context omitted.

Even if you could fit a 500B model's expert weights in very fast system RAM, it would run so slow as to be useless.

That's really only "useless" if the only thing you care about is a quick real-time response. Contrary to common perception, MoE models do benefit from batching requests together even when run on a single node, you just have to ensure you have at least ~5 parallel requests in flight (and that's for the very sparsest models) to really see the aggregate benefit. (Intuitively, that's because the issue of whether any acti…

This is just wishful thinking.

For prefill, it's really easy to batch MoE and get really good tk/s, even on a single stream.

For decode, you will run into the problem that:

1) you need more parallel requests which means more memory for context

2) 5 requests will not give you very much expert overlap on parallel requests

Re: Memory has grown to nearly two-thirds of AI chip component costs

#418

Post by zozbot234 »

EnPissant wrote: post #417
zozbot234 wrote: post #412

Earlier quoted context omitted.

That's really only "useless" if the only thing you care about is a quick real-time response. Contrary to common perception, MoE models do benefit from batching requests together even when run on a single node, you just have to ensure you have at least ~5 parallel requests in flight (and that's for the very sparsest models) to really see the aggregate benefit. (Intuitively, that's because the issue of whether any acti…

This is just wishful thinking. For prefill, it's really easy to batch MoE and get really good tk/s, even on a single stream. For decode, you will run into the problem that: 1) you need more parallel requests which means more memory for context 2) 5 requests will not give you very much expert overlap on parallel requests

You don't need "very much" expert overlap to see aggregate gains at scale, you just need some of it; that's where the "birthday" framing becomes relevant. Memory for context is an issue, but recent models like DeepSeek V4 use very little of it even at relatively large contexts.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#419

Post by EnPissant »

zozbot234 wrote: post #418
EnPissant wrote: post #417

Earlier quoted context omitted.

This is just wishful thinking. For prefill, it's really easy to batch MoE and get really good tk/s, even on a single stream. For decode, you will run into the problem that: 1) you need more parallel requests which means more memory for context 2) 5 requests will not give you very much expert overlap on parallel requests

You don't need "very much" expert overlap to see aggregate gains at scale, you just need some of it; that's where the "birthday" framing becomes relevant. Memory for context is an issue, but recent models like DeepSeek V4 use very little of it even at relatively large contexts.

>You don't need "very much" expert overlap to see aggregate gains at scale, you just need some of it

I'm not sure what you are claiming. Decode is bottle-necked by memory bandwidth. To see a speed up of 2x, you have to ensure each expert weight memory fetch can be used by 2 parallel streams. What exactly is the average factor you are claiming for 5x parallel streams (due to "birthday paradox" factors)? The Birthday paradox isn't really relevant here. It's about coverage, not parallelism.

> Memory for context is an issue, but recent models like DeepSeek V4 use very little of it even at relatively large contexts.

This is not true.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#420

Post by willis936 »

gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

This line of thinking makes sense if we're talking about opex like power usage. This is capex though and we'll be financing this overpaying for a long time after the hardware has "aged out". Not really sure there is an upside to it.

Also, inference cost predictions were made before this price jump, so we really haven't started paying for it yet. Inference will not be getting cheaper.

Post reply on HN