Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory has grown to nearly two-thirds of AI chip component costs

epoch.ai

Post reply on HN
Print view View original

131–140 of 524 posts

Re: Memory has grown to nearly two-thirds of AI chip component costs

#131

Post by forrestthewoods »

alphabeta3r56 wrote: post #68
forrestthewoods wrote: post #62

Earlier quoted context omitted.

if you can buy one! The RTX 5090 is faster than an H200. It just has less ram (32 vs 141), doesn't have NVLink, and technically isn't allowed to be used in a datacenter. The datacenter GPUs sell at an 80% margin. They're incredibly overpriced. But the laws of supply and demand are undefeated and so here we all are.

> The RTX 5090 is faster than an H200. It just has less ram H200 has HBM and much more 64-bit compute

Let me try again.

RTX 5090 has more CUDA cores that run at a higher clock speed. H200 has more RAM and significantly more RAM bandwidth.

Which one is net faster depends on your use case. But you may be very surprised that many workflows are faster on an RTX 5090!

Re: Memory has grown to nearly two-thirds of AI chip component costs

#132

Post by chinathrow »

trollbridge wrote: post #118
slicktux wrote: post #2

I bought 96GB of RAM a couple of years ago for ~$250. That same RAM now costs $1200!

I bought 192GB of DDR3 a year ago for literally $60 ($5 a stick). It's about $22 a stick now, so more like $350 today. What on earth is _anybody_ doing with DDR3?

Being desperate?

Re: Memory has grown to nearly two-thirds of AI chip component costs

#133

Post by manquer »

trollbridge wrote: post #118
slicktux wrote: post #2

I bought 96GB of RAM a couple of years ago for ~$250. That same RAM now costs $1200!

I bought 192GB of DDR3 a year ago for literally $60 ($5 a stick). It's about $22 a stick now, so more like $350 today. What on earth is _anybody_ doing with DDR3?

All memory products use many shared resources in the supply chain, so if there is high demand in one product line, others have to raise prices to compete for the resources or stop making those lines altogether.

That is to say at least you were able to buy them at $350 today, with the current trajectory there will be no supply at all in few months.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#134

Post by aleph_minus_one »

tjwebbnorfolk wrote: post #115
nicoburns wrote: post #110

Earlier quoted context omitted.

Because fabs are about the most complex cutting edge technology out there: the "rocket science" of our day (or one of them). And merely having the money is not sufficient. It would be very easy to blow several billion dollars and end up with nothing to show for it. Just look at how Intel has struggled to compete in recent years, and they have been in the business for decades.

Intel struggled because they bet the company that Moore's law was over back in ~2014, and instead of upgrading their fabs to EUV they sent the money back to shareholders. They forgot Moore's main lesson: only the paranoid survive. They thought they could coast, and it nearly killed them.

> They forgot Moore's main lesson: only the paranoid survive.

"Only the Paranoid Survive" is rather a quote and book title by Andrew S. Grove.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#135

Post by echoangle »

gruez wrote: post #92
johnvanommen wrote: post #69

Earlier quoted context omitted.

Possibly the best deal there is I really need to shut up, or bite the bullet and by one. If you graph the tokens per second on the 5090, your jaw will hit the floor at how cheap it is

With only 32gb of vram, you can only run small/quantized models, in which case what's the point? At $4000, that gets you 20 months of 10x claude or chagpt subscriptions, which provide far better models. You'd need some use case where you can tolerate worse models, and use a steady supply of them. That doesn't match most people's usage patterns.

Or you want to process private data or don’t have reliable connectivity. There are a few more reasons for local models I think.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#136

Post by jlokier »

trollbridge wrote: post #118
slicktux wrote: post #2

I bought 96GB of RAM a couple of years ago for ~$250. That same RAM now costs $1200!

I bought 192GB of DDR3 a year ago for literally $60 ($5 a stick). It's about $22 a stick now, so more like $350 today. What on earth is _anybody_ doing with DDR3?

Demand for DDR3 is up because people who want DDR5 or DDR4 but can't afford either any more are choosing DDR3 and old DDR3-compatible systems to put it in, instead of what they really want.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#138

Post by thfuran »

DoctorOetker wrote: post #105
mchusma wrote: post #41

Everything I read seems to suggest that RAM capacity is going to grow at 20-25% a year, which just doesn't seem good enough. Even in consumer use cases, phones and laptops would benefit greatly by double the amount of RAM. And then obviously, the AI need is gigantic. I don't see it going away. I mean, it may not grow as fast as now, but I don't see it growing away either. I get why the memory makers do not want to ba…

According to the recent article HBM memory is 3x less efficient wafer area wise than LPDDR; but the bandwidth is more than triple. What if its in everyone's interest to buy computers at say 1/3rd the rate and switch everything over to HBM? the discrepancy between compute and memory has been growing for ages, perhaps a painful switch to HBM is exactly what we need? Would you rather have 3 intermediate computers with l…

Not many workloads are RAM bandwidth limited. Power and latency are much more common bottlenecks, and HBM loses on both of those.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#139

Post by Npovview »

elorant wrote: post #8

Bought a second hand Dell server a week ago. The entire rig with a 12-core CPU and 32GB DDR4 ecc RAM cost as much as I'd pay to buy 64 GB of DDR RAM alone. I hope there's an end to this absurdity soon enough otherwise the pain will affect other markets too. I read the other day that PC case sales have collapsed by more than 40%.

I have an alternative take.

If hyperscalers are using more RAM, and that RAM is not available for consumers, it means all the heavy stuff will happen in the cloud. Why would we want both the hyperscalers and consumers to have RAM simultaneously? Consumers would want more RAM to run local models but then hyperscalers capacity will be unused.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#140

Post by Galanwe »

johnvanommen wrote: post #69
layer8 wrote: post #67

Earlier quoted context omitted.

An unbelievably good deal at $4000 plus?

Possibly the best deal there is I really need to shut up, or bite the bullet and by one. If you graph the tokens per second on the 5090, your jaw will hit the floor at how cheap it is

The 5090 is crap for inference. Unless you like dummy models, sure they will run at light speed. All the rage is MoE with 500B-1T weights nowadays.
Post reply on HN