Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory has grown to nearly two-thirds of AI chip component costs

epoch.ai

Post reply on HN
Print view View original

231–240 of 524 posts

Re: Memory has grown to nearly two-thirds of AI chip component costs

#231

Post by ahartmetz »

esseph wrote: post #192
brandensilva wrote: post #183

Earlier quoted context omitted.

China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply. But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer. I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.

I suspect Chinese factories will get built first, but quality may take a few years to really nail down. Basically: China floods the market with cheaper but less QA'd parts, makes a gazillion dollars, is able to spend said money to fix yields / QA issues and streamline operations, by the time that happens Micron and maybe a few other existing players will have new memory production, and then we'll have a flood of chea…

It is not a law of nature that Chinese products are lower quality (cf. electric cars) and I don't see why they would go for that. They can just bin what they produce like everyone else and sell their products for what they have been tested to deliver.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#232

Post by helterskelter »

throwatdem12311 wrote: post #165
ecommerceguy wrote: post #137

As models gain efficiency, will the need for ram cool?

They’ll just fill up the ram with bigger models. Demand will INCREASE, not decrease.

Every time we add capacity with almost anything, we find ways to saturate it.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#233

Post by zozbot234 »

Galanwe wrote: post #140
johnvanommen wrote: post #69

Earlier quoted context omitted.

Possibly the best deal there is I really need to shut up, or bite the bullet and by one. If you graph the tokens per second on the 5090, your jaw will hit the floor at how cheap it is

The 5090 is crap for inference. Unless you like dummy models, sure they will run at light speed. All the rage is MoE with 500B-1T weights nowadays.

MoE is fine. You can put the shared weights on the 5090 (will fit handily even for the largest models) and expert weights on CPU, possibly with weights offload from storage.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#234

Post by cyost »

esseph wrote: post #192
brandensilva wrote: post #183

Earlier quoted context omitted.

China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply. But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer. I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.

I suspect Chinese factories will get built first, but quality may take a few years to really nail down. Basically: China floods the market with cheaper but less QA'd parts, makes a gazillion dollars, is able to spend said money to fix yields / QA issues and streamline operations, by the time that happens Micron and maybe a few other existing players will have new memory production, and then we'll have a flood of chea…

They're doing decent enough already for consumer electronics. Corsair is selling 16GB 6000MT/s CL36 DDR5 sticks in China using memory from CXMT: https://www.tomshardware.com/pc-components/ddr5/chinese-memo...

Re: Memory has grown to nearly two-thirds of AI chip component costs

#235

Post by helterskelter »

Legend2440 wrote: post #12
positron26 wrote: post #3

The algorithm advances are going to crash this so hard.

Or will more efficient algorithms just mean we run even more AI models, increasing the demand for AI chips even more?

Better algos = more demand

Memory squeeze will get worse before it gets better.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#236

Post by Closi »

aDyslecticCrow wrote: post #177
sandworm101 wrote: post #147

Earlier quoted context omitted.

Supply will not meet demand. What incentive do the handful of dram manufacturers have to end the party? This is what happens when legal monopolies finally win control. Dont't worry. The patents will expire in a few decades. Our grandkids will see DDR5 get cheap again. The system functions as intended.

Patents is not the issue here. Not even close. The up-front investment of a memory fab is measured in billions, and takes years to construct and get running. The margin on the chips themselves is terrible, so without scale its not worth even trying. DDR5 is a industry standard that takes some effort to conform to, but the licence fees is a drop in the bucket to the cost of creating a fab. The fabricators were cautiou…

Billions is nothing in this market - if the market is supply constrained in the medium term then the hyperscalers will purchase their own route to manufacture (e.g. through coinvestment).

Also that's not what the bullwhip effect is - although I know what you are saying. The bullwhip scenario is about the effect of communication and batching through various layers in the supply chain, this is more similar to the cobweb effect/theory.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#237

Post by gruez »

regularfry wrote: post #182
gruez wrote: post #92

Earlier quoted context omitted.

With only 32gb of vram, you can only run small/quantized models, in which case what's the point? At $4000, that gets you 20 months of 10x claude or chagpt subscriptions, which provide far better models. You'd need some use case where you can tolerate worse models, and use a steady supply of them. That doesn't match most people's usage patterns.

If you can do what you need with qwen3.6-27b, it starts to look really interesting. That model is crazy good for the size, but it's a pain tweaking the params to run it on a 4090 with decent context and decent token speed. A 5090 looks tasty from that point of view, and only more so if you think in terms of the probability of that model being roflstomped by something in the same weight class in the next couple of yea…

>If you can do what you need with qwen3.6-27b, it starts to look really interesting.

What's the use case here? Churning out massive amounts of slop code through autonomous agents? Running openclaw 24/7? I think the proliferation of codex and claude code, compared to any of the cheaper open models suggests that at least for most software development, the 50-75% discount of open models isn't worth the hassle of the decreased intelligence.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#238

Post by Closi »

AlotOfReading wrote: post #223
fitblipper wrote: post #164

Earlier quoted context omitted.

I have fairly simplistic view of the economics involved here. Could you explain why the ability to sell more chips wouldn't be sufficient enough incentive to increase supply?

Let's imagine you're drilling oil instead. You have to spend billions of dollars over years finding and developing a new oilfield to make any profit back. And once you have it, you have to continuously spend enormous amounts of money to keep producing it, which means your effective price floor is higher than the current stable price. Now it's 2021 and someone gets a tanker stuck in the Suez, sending the price of oil…

Although on the flipside, let's pretend it's 2017's and you are Nvidia selling GPU's for Bitcoin - maybe demand will dry up at some point? Do you stop scaling production as this might be the max of the market, or do you follow the market and increase production?

It's always easier to see the right move in hindsight!

Re: Memory has grown to nearly two-thirds of AI chip component costs

#239

Post by joha4270 »

pastel8739 wrote: post #166
thfuran wrote: post #138

Earlier quoted context omitted.

Not many workloads are RAM bandwidth limited. Power and latency are much more common bottlenecks, and HBM loses on both of those.

Isn’t memory bandwidth super relevant for AI?

Yes and so we use HBM for AI (among other things), but that's an exception. For things like games or displaying webpages, its not very important and we generally don't put HBM into things for that.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#240

Post by ahartmetz »

pastel8739 wrote: post #166
thfuran wrote: post #138

Earlier quoted context omitted.

Not many workloads are RAM bandwidth limited. Power and latency are much more common bottlenecks, and HBM loses on both of those.

Isn’t memory bandwidth super relevant for AI?

It is like the most important performance figure. When I use an LLM that mostly fits on my GPU, the GPU will run at about 30% of its maximum power consumption - probably because the memory can't feed the ALUs fast enough. Similarly for the part that runs on the CPU, the CPU cores will show 100% utilization but not consume as much power as they usually do under full load. The GUI will also be choppier than usual under full load (noticeable, but not too annoying) presumably because pixel pushing also needs some nontrivial memory bandwidth which is hard to get.
Post reply on HN