Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory has grown to nearly two-thirds of AI chip component costs

epoch.ai

Post reply on HN
Print view View original

181–190 of 524 posts

Re: Memory has grown to nearly two-thirds of AI chip component costs

#181

Post by array_key_first »

ec109685 wrote: post #179
radialstub wrote: post #161

Earlier quoted context omitted.

The memory makers will not expand demand drastically. It is in the nature of their business to keep the market under-supplied, otherwise the following oversupply will kill them. Instead, supply is just rerouted from less profitable segments such as mobile and personal computing.

Supply and demand always balance out. There is no way manufacturers aren’t going to compete away these inflated margins, as long as they feel like this demand is sustainable.

There's very few manufacturers, I believe 3 globally? And there's a large moat. Nobody can compete with them in the next 10 years. It's really not hard to coordinate action between 3 companies.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#182

Post by regularfry »

gruez wrote: post #92
johnvanommen wrote: post #69

Earlier quoted context omitted.

Possibly the best deal there is I really need to shut up, or bite the bullet and by one. If you graph the tokens per second on the 5090, your jaw will hit the floor at how cheap it is

With only 32gb of vram, you can only run small/quantized models, in which case what's the point? At $4000, that gets you 20 months of 10x claude or chagpt subscriptions, which provide far better models. You'd need some use case where you can tolerate worse models, and use a steady supply of them. That doesn't match most people's usage patterns.

If you can do what you need with qwen3.6-27b, it starts to look really interesting. That model is crazy good for the size, but it's a pain tweaking the params to run it on a 4090 with decent context and decent token speed. A 5090 looks tasty from that point of view, and only more so if you think in terms of the probability of that model being roflstomped by something in the same weight class in the next couple of years. I reckon that probability is significantly non-zero, but fundamentally it's a guess.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#183

Post by brandensilva »

radialstub wrote: post #161
gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

The memory makers will not expand demand drastically. It is in the nature of their business to keep the market under-supplied, otherwise the following oversupply will kill them. Instead, supply is just rerouted from less profitable segments such as mobile and personal computing.

China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply.

But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer.

I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#184

Post by [deleted] »

throwatdem12311 wrote: post #159
aunty_helen wrote: post #66

Earlier quoted context omitted.

This is 100% going to kill the home built pc market. When I started building gaming pcs, the top top card was 750$ (NZD). Now they’re 10,000 just for the gpu and another 1-2000 for ram. People used to get into gaming pcs as an affordable hobby, now it’s making general aviation look like plan B.

Don’t you worry - Microsoft and Amazon will have you covered with cloud streaming. Can’t afford a computer because they bought up all the supply? They’ll conveniently sell it back to you with a subscription! You’ll own nothing and be happy.

[deleted]

Re: Memory has grown to nearly two-thirds of AI chip component costs

#185

Post by luqtas »

aunty_helen wrote: post #66
oceansky wrote: post #6

Awful time for gamers and PC hobbyists not fully into AI.

This is 100% going to kill the home built pc market. When I started building gaming pcs, the top top card was 750$ (NZD). Now they’re 10,000 just for the gpu and another 1-2000 for ram. People used to get into gaming pcs as an affordable hobby, now it’s making general aviation look like plan B.

there's much more than triple A video-games running at 240 Hz on Ultra settings... a 200 USD laptop/computer has enough power to run hundreds of interesting indie games and AAA from the past

Re: Memory has grown to nearly two-thirds of AI chip component costs

#186

Post by stevenwoo »

DoctorOetker wrote: post #30

It's still unclear to me: the shortage is semiconductor boules / wafers? or the shortage is semiconductor fab process step availability? As long as the discussion seems focused on memory, I'd suspect the latter, but if its really the semiconductor boules/wafers, then I'd expect the boule growers to profit, not the memory makers, who just pass on the cost. So which is it?

This covers it pretty well https://news.ycombinator.com/item?id=48229319, TLDR -memory for AI uses more wafers from same production line as other memory and is more profitable, building new fab very risky historically for companies. The companies have cut production of other memory to favor memory for AI and the market for memory for AI is still unfulfilled so prices still go up for customers of every type.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#187

Post by treis »

Legend2440 wrote: post #31

I wonder why the hyperscalers aren't vertically integrating more and building their own fabs. Sure, a fab costs a billion dollars, but they're currently spending hundreds of billions of dollars purchasing chips from NVidia and others.

A fab costs a billion dollars (really a lot more) and 5 years. It doesn't do anything for anyone today.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#188

Post by roxolotl »

overfeed wrote: post #171
gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

It sure looks like Sam Altman's masterful gambit to corner the memory market has had unforeseen consequences.

Is any of this actually unforeseen? Buying the vast majority of the world’s supply of something does have mostly predictable consequences.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#189

Post by regularfry »

DoctorOetker wrote: post #30

It's still unclear to me: the shortage is semiconductor boules / wafers? or the shortage is semiconductor fab process step availability? As long as the discussion seems focused on memory, I'd suspect the latter, but if its really the semiconductor boules/wafers, then I'd expect the boule growers to profit, not the memory makers, who just pass on the cost. So which is it?

Regardless of the specific mechanics of the bottleneck, we know what the proximate source of the problem is: openai locking up 40% of Samsung and SK Hynix wafer capacity for the next few years. That's what triggered the madness.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#190

Post by kristopolous »

ec109685 wrote: post #179
radialstub wrote: post #161

Earlier quoted context omitted.

The memory makers will not expand demand drastically. It is in the nature of their business to keep the market under-supplied, otherwise the following oversupply will kill them. Instead, supply is just rerouted from less profitable segments such as mobile and personal computing.

Supply and demand always balance out. There is no way manufacturers aren’t going to compete away these inflated margins, as long as they feel like this demand is sustainable.

You know there's other strategies? Companies can be more clever than naively undercutting each other...

Memory in particular ... https://en.wikipedia.org/wiki/DRAM_price_fixing_scandal

The entry-cost to getting into memory is on the order of $billions and years - you can do just about anything...

Post reply on HN