Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory has grown to nearly two-thirds of AI chip component costs

epoch.ai

Post reply on HN
Print view View original

241–250 of 524 posts

Re: Memory has grown to nearly two-thirds of AI chip component costs

#241

Post by tooltalk »

radialstub wrote: post #161
gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

The memory makers will not expand demand drastically. It is in the nature of their business to keep the market under-supplied, otherwise the following oversupply will kill them. Instead, supply is just rerouted from less profitable segments such as mobile and personal computing.

This is wrong. It is NOT in their nature to keep the market under-supplied -- eg, Samsung, the industry's largest company, was notorious for expanding their capacity during the industry downturn to gain market share while everyone else was cutting back to minimize loss.

I'm guessing you are also probably unfamiliar with the terms like "chicken game" which refers to the cutthroat, high-stakes price wars where dominant semiconductor manufacturers intentionally overproduce and slash prices. This is literally how the industry went from dozens to just three majors today since the 80's.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#242

Post by nomel »

ahartmetz wrote: post #231
esseph wrote: post #192

Earlier quoted context omitted.

I suspect Chinese factories will get built first, but quality may take a few years to really nail down. Basically: China floods the market with cheaper but less QA'd parts, makes a gazillion dollars, is able to spend said money to fix yields / QA issues and streamline operations, by the time that happens Micron and maybe a few other existing players will have new memory production, and then we'll have a flood of chea…

It is not a law of nature that Chinese products are lower quality (cf. electric cars) and I don't see why they would go for that. They can just bin what they produce like everyone else and sell their products for what they have been tested to deliver.

But it is a near law that the first to market attempts will fully embraces the deeply engrained culture of 差不多, until market forces beat it out of the product line.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#243

Post by plipt »

regularfry wrote: post #189
DoctorOetker wrote: post #30

It's still unclear to me: the shortage is semiconductor boules / wafers? or the shortage is semiconductor fab process step availability? As long as the discussion seems focused on memory, I'd suspect the latter, but if its really the semiconductor boules/wafers, then I'd expect the boule growers to profit, not the memory makers, who just pass on the cost. So which is it?

Regardless of the specific mechanics of the bottleneck, we know what the proximate source of the problem is: openai locking up 40% of Samsung and SK Hynix wafer capacity for the next few years. That's what triggered the madness.

Is there an understanding of what OpenAI intends to do with that memory?

Surely they need GPU capacity and would need memory for those GPUs but OpenAI doesn't build GPUs or any hardware, right? So did they pay to keep the supply locked up, or do they have the ability to put that ram into use?

Re: Memory has grown to nearly two-thirds of AI chip component costs

#244

Post by alecco »

epistasis wrote: post #60
Legend2440 wrote: post #31

I wonder why the hyperscalers aren't vertically integrating more and building their own fabs. Sure, a fab costs a billion dollars, but they're currently spending hundreds of billions of dollars purchasing chips from NVidia and others.

I'm not sure if they should vertically integrate, it would probably be a better idea to directly fund the expansion of capacity, much like Apple does when they scale up a new technology for iPhones. However, that the hyperscalers and AI companies aren't doing this says a lot about their true beliefs about how much future demand AI will have. AI companies claim they will need a ton of massive expansion, but are unwill…

> [...] better idea to directly fund the expansion of capacity [...] > > However, that the hyperscalers and AI companies aren't doing this says a lot about their true beliefs about how much future demand AI will have.

With what money? They have to spend the money they get on hardware ASAP else they are left behind.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#245

Post by eldenring »

gpm wrote: post #153
eldenring wrote: post #143

Earlier quoted context omitted.

2-3x is completely dwarfed by the remaining improvements in training which is still in its infancy relatively

Probably, but at some point we're very likely to run out of significant training improvements and it's not clear that we'll see that point coming from a long way out. Likewise it's probably dwarfed by improvements in how we make dram - continuing the roughly exponential (maybe a bit less recently) scaling of chips - but not necessarily. The 2x from returning to previous costs is interesting because it's practically g…

my reply from the other thread fits here too:

> this is just not true at all, there are massive leaps from algorithms, data, etc. every year. scale is one axis of many and you need to get them all correct.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#246

Post by Keyframe »

jlokier wrote: post #136
trollbridge wrote: post #118

Earlier quoted context omitted.

I bought 192GB of DDR3 a year ago for literally $60 ($5 a stick). It's about $22 a stick now, so more like $350 today. What on earth is _anybody_ doing with DDR3?

Demand for DDR3 is up because people who want DDR5 or DDR4 but can't afford either any more are choosing DDR3 and old DDR3-compatible systems to put it in, instead of what they really want.

At the rate we're going, soon we're going to draw from SIMM stock.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#247

Post by XorNot »

brandensilva wrote: post #183
radialstub wrote: post #161

Earlier quoted context omitted.

The memory makers will not expand demand drastically. It is in the nature of their business to keep the market under-supplied, otherwise the following oversupply will kill them. Instead, supply is just rerouted from less profitable segments such as mobile and personal computing.

China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply. But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer. I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.

China wants a sovereign DRAM capacity. They're playing an entirely different game then the commercial suppliers in the West.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#248

Post by dev1ycan »

radialstub wrote: post #161
gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

The memory makers will not expand demand drastically. It is in the nature of their business to keep the market under-supplied, otherwise the following oversupply will kill them. Instead, supply is just rerouted from less profitable segments such as mobile and personal computing.

CXMT is scaling up incredibly fast, they are on a clock (south koreans) their monopoly will end relatively soon, although I'm guessing that the AI companies will crash before that anyways.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#249

Post by Fnoord »

jayd16 wrote: post #198
luqtas wrote: post #185

Earlier quoted context omitted.

there's much more than triple A video-games running at 240 Hz on Ultra settings... a 200 USD laptop/computer has enough power to run hundreds of interesting indie games and AAA from the past

Yeah sure, but some folks were in it for the hot rods too.

Were in for the hot rods.

You can still play fantastic games with amazing gameplay, great storytelling, and even requiring quite a GPU. But you won't upgrade your GPU or RAM. If it gets broken, people have already gotten their money back instead of replacement (whether that is legal or not, depends on your jurisdiction, and regardless: it is happening). So the demand and adoption of say 240 Hz 4k OLED gaming is going to slow. I currently sport two 1440p IPS capable of 144 Hz, with an AMD 6700 XT, 64 GB DDR4, and a 5700X3D. I'll wait upgrading that to a 4k rig.

What I will do is buy a Nintendo Switch 2 before the price increase hits. Why? Great gameplay for kids.

Post reply on HN