Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory has grown to nearly two-thirds of AI chip component costs

epoch.ai

Post reply on HN
Print view View original

161–170 of 524 posts

Re: Memory has grown to nearly two-thirds of AI chip component costs

#161

Post by radialstub »

gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

The memory makers will not expand demand drastically. It is in the nature of their business to keep the market under-supplied, otherwise the following oversupply will kill them. Instead, supply is just rerouted from less profitable segments such as mobile and personal computing.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#162

Post by Auracle »

irthomasthomas wrote: post #86
johnvanommen wrote: post #56

Earlier quoted context omitted.

Nine years after Google's seminal paper lit the fuse on AI, a total lack of manufacturing foresight has trapped over a trillion dollars of incoming capital in a hardware bottleneck. The entire sector is now facing a critical RAM starvation crisis where memory manufacturers are actively slow-rolling supply just to keep prices high and avoid running out entirely. This has created an unprecedented supply-and-demand dist…

A lot of words to say that Sam Altman bought up the worlds total supply of ram chips for the next few years.

A dick move or just really prescient?

Re: Memory has grown to nearly two-thirds of AI chip component costs

#164

Post by fitblipper »

sandworm101 wrote: post #147
gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

Supply will not meet demand. What incentive do the handful of dram manufacturers have to end the party? This is what happens when legal monopolies finally win control. Dont't worry. The patents will expire in a few decades. Our grandkids will see DDR5 get cheap again. The system functions as intended.

I have fairly simplistic view of the economics involved here. Could you explain why the ability to sell more chips wouldn't be sufficient enough incentive to increase supply?

Re: Memory has grown to nearly two-thirds of AI chip component costs

#166

Post by pastel8739 »

thfuran wrote: post #138
DoctorOetker wrote: post #105

Earlier quoted context omitted.

According to the recent article HBM memory is 3x less efficient wafer area wise than LPDDR; but the bandwidth is more than triple. What if its in everyone's interest to buy computers at say 1/3rd the rate and switch everything over to HBM? the discrepancy between compute and memory has been growing for ages, perhaps a painful switch to HBM is exactly what we need? Would you rather have 3 intermediate computers with l…

Not many workloads are RAM bandwidth limited. Power and latency are much more common bottlenecks, and HBM loses on both of those.

Isn’t memory bandwidth super relevant for AI?

Re: Memory has grown to nearly two-thirds of AI chip component costs

#167

Post by sowbug »

DoctorOetker wrote: post #55
AnotherGoodName wrote: post #42

Earlier quoted context omitted.

It’s fab capacity. Fwiw dram is different enough that fabs are not transferable between dram memory and other usages. It’s nice to think ‘wow if they made the current 10nm dram on the latest 2nm processes it’d be much faster’ but it doesn’t work that way. The specific size is needed for the capacitance. Sram can be made on fabs that make other circuitry since it’s transistor not capacitor based but is less dense. Dra…

I know the differences between SRAM, DRAM, ... I asked for evidence different people keep feeding me opposite stories: one insists its not fab capacity but wafer competition, with a recent article claiming HBM3E takes 3 times as much wafer area per bit than LPDDR5X. Others tell me the complete opposite: its fab capacity, not wafer shortage. Do we have citable references to ground either set of claims?

I believe those are two ways of describing the same thing. If you're able to book some fab capacity, that means you get to decide what the fab does with the next wafers in the queue.

From your sibling comment, I think you're interpreting the 3x HBM stat as contributing to making wafers scarce. It's more that the next wafer to be processed in a fab is especially precious, making the opportunity cost larger. The beach sand remains plentiful.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#168

Post by jmalicki »

Waterluvian wrote: post #121
gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

What’s the lifespan/refurbishability of the capex elements like the “GPU” modules or even the DRAM soldered into them?

For lifespan, AWS is still running a ton of T4 GPUs from 2018, that power a lot of computer vision models. A ton of these will have a long life, not all ML is about frontier LLMs.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#169

Post by mrandish »

johnvanommen wrote: post #72
KronisLV wrote: post #20

I'm not moving past my DDR4 build (and the 32 GB of DDR4 2133 MHz backup chips I still have around from way back, before I got the current 3200 MHz ones) until the prices go back to being at least partially sane. This also means that CPU manufacturers are not getting my money (since the 5800X is fine for now) and I have no reason to get a new GPU either (though admittedly the B580 isn't perfect).

What if this is the lowest that prices will ever be?

As Yogi Berra famously said, "It's tough to make predictions, especially about the future." But based on historical tech industry trends, a price increase in one component that's this rapid and extreme, is likely to eventually regress somewhat toward the long-term trend line - even if that trend line experiences a longer-term shift upward.

As always, some interpret certain recent events as reason to conclude "but this time it's different." Occasionally they are correct. But that doesn't change the fact that it's reasonable to assume some of the recent extreme, rapid price inflation is due to shorter term market distortion. It's also pretty clear that some of the recent increase in demand represents a stable increase in the long-term trendline. The question is how much is long-term stable and how much is short-term distortion.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#170

Post by epistasis »

energy123 wrote: post #124
epistasis wrote: post #60

Earlier quoted context omitted.

I'm not sure if they should vertically integrate, it would probably be a better idea to directly fund the expansion of capacity, much like Apple does when they scale up a new technology for iPhones. However, that the hyperscalers and AI companies aren't doing this says a lot about their true beliefs about how much future demand AI will have. AI companies claim they will need a ton of massive expansion, but are unwill…

Oracle is getting sold because of how much capex they're spending on new data centers in the middle of a high rates environment. It's not like they're stockpiling cash due to doubting AI.

Oracle had not entered into my thoughts at all; I know they do some cloud stuff but they are in a very different position than OpenAI or Anthropic or Google.
Post reply on HN