Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory has grown to nearly two-thirds of AI chip component costs

epoch.ai

Post reply on HN
Print view View original

141–150 of 524 posts

Re: Memory has grown to nearly two-thirds of AI chip component costs

#141

Post by cogman10 »

ksec wrote: post #29
slicktux wrote: post #2

I bought 96GB of RAM a couple of years ago for ~$250. That same RAM now costs $1200!

It is one of the thing with consumer when they remember they brought it at the absolutely lowest price point when DRAM maker were bleeding money. Those are not normal pricing. Before the pricing collapse in early 2020, 96GB DDR5 would have cost about $450 to $500. And I will need to restate again the cost of DRAM hasn't really changed much in the past 20 years. Its price just goes up and down in cycles. So in reality…

> Its price just goes up and down in cycles.

Historically the price has always trended downward. When I first got into computing $200 could buy you 128 MB (yes M) of ram. Really nice systems had 512 MB.

That's obviously changed over the decades as process shrinks have lead to higher memory density. We should generally expect that ram will cheaper up and until the point where process shrinks stop happening. They've definitely slowed, but they haven't stopped.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#143

Post by eldenring »

gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

2-3x is completely dwarfed by the remaining improvements in training which is still in its infancy relatively

Re: Memory has grown to nearly two-thirds of AI chip component costs

#144

Post by Forgeties79 »

dawnerd wrote: post #80
slicktux wrote: post #2

I bought 96GB of RAM a couple of years ago for ~$250. That same RAM now costs $1200!

I’m so mad I didn’t max out my main server when I had the chance. Used enterprise sticks were dirt cheap on eBay.

Used enterprise HDD’s also jacked up now. It’s absurd lol

Re: Memory has grown to nearly two-thirds of AI chip component costs

#145

Post by moregrist »

bushbaba wrote: post #54
slicktux wrote: post #2

I bought 96GB of RAM a couple of years ago for ~$250. That same RAM now costs $1200!

Makes prior assumptions that getting tens of gigs of ram is cheap thrown out the window. Would likely lead to super fast SSDs such as optain being way more valuable

The price of SSDs is similarly depressing.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#146

Post by minraws »

blululu wrote: post #128
minraws wrote: post #64

Earlier quoted context omitted.

I mean the biggest risk is Chinese CXML benefits and capturing markets that others are leaving hanging and then being able to compete and push out the others when costs start to normalize. As for 20-25% growth not being enough, I think it's not that far off, if we assume data center build out plans hit a wall and slow down significantly, and the AI heat starts to cool off. I don't think 20-25% may be enough in the sh…

Looking at the history of the memory industry the biggest risk is that a firm would over produce and go bankrupt. Maybe this time is different but so far no memory chip maker has gone under because their competition increased capacity.

I might be wrong but your second point can't be true if the first one is true.

Let me explain, imagine CXML grows massive and builds a lot of fabs, so much so that it becomes the leader in multiple segments, then the market demand cools off.

Then CXML the company that invested massively has oversupply so it undercuts every other memory company.

Aka, Samsung, SK Hynix are dead, and to protect Micron now US has 10000% tariff on the supply of memory.

Imagine. Because that has happened, if you don't play the boom and bust game someone will because the market is very large during a boom, and generally the player scaling more isn't the one with margins to protect and generally has the ability to undercut others.

Asian memory chip giants were made by under cutting European and American companies, American companies adapted by moving manufacturing to Asia, and European ones got bought for pennies or dissolved.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#147

Post by sandworm101 »

gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

Supply will not meet demand. What incentive do the handful of dram manufacturers have to end the party? This is what happens when legal monopolies finally win control. Dont't worry. The patents will expire in a few decades. Our grandkids will see DDR5 get cheap again. The system functions as intended.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#148

Post by baq »

paulmist wrote: post #112
oceansky wrote: post #6

Awful time for gamers and PC hobbyists not fully into AI.

I think it's the opposite. Sure in short term hobbyists are getting squeezed, but the amount of capital that they can put into pushing the edge is small compared to Fortune 500. Sooner or later hobbyists will benefit, especially if the market crashes.

If it crashes after it kills the PC we’ll be left with… nothing? Path matters as much as destination

Re: Memory has grown to nearly two-thirds of AI chip component costs

#149

Post by cubefox »

gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

For some reason I still haven't heard any predictions on when new fabs will come online to meet the current demand. This shouldn't be too hard to find out, since the building time of fabs is very predictable process.

The difficult question is more whether foreseeable memory demand will remain at the current level, grow even further, or shrink again.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#150

Post by EnPissant »

gruez wrote: post #92
johnvanommen wrote: post #69

Earlier quoted context omitted.

Possibly the best deal there is I really need to shut up, or bite the bullet and by one. If you graph the tokens per second on the 5090, your jaw will hit the floor at how cheap it is

With only 32gb of vram, you can only run small/quantized models, in which case what's the point? At $4000, that gets you 20 months of 10x claude or chagpt subscriptions, which provide far better models. You'd need some use case where you can tolerate worse models, and use a steady supply of them. That doesn't match most people's usage patterns.

Also, electricity isn't free.
Post reply on HN