if we survive the bubble bursting and there isn't a "too big to fail" bailout with public money manipulation by bought politicians
we are going to have amazing cheap used hardware for a decade
Memory has grown to nearly two-thirds of AI chip component costs
101–110 of 524 posts
Re: Memory has grown to nearly two-thirds of AI chip component costs
#102I bought 96GB of RAM a couple of years ago for ~$250. That same RAM now costs $1200!
I bought a couple of used computers with 256 GB of DDR 4 (total) a year ago. The ram is worth more than I paid for the whole machines now.
Re: Memory has grown to nearly two-thirds of AI chip component costs
#103I bought 96GB of RAM a couple of years ago for ~$250. That same RAM now costs $1200!
Ramflation
Re: Memory has grown to nearly two-thirds of AI chip component costs
#104It seems to me the max memory you can buy in a laptop stagnated for the past 3 years or so.
I have always felt insulted that most laptops even offer a low 4 GB of RAM I rather take 16 GB in previous gen memory
Re: Memory has grown to nearly two-thirds of AI chip component costs
#105Everything I read seems to suggest that RAM capacity is going to grow at 20-25% a year, which just doesn't seem good enough. Even in consumer use cases, phones and laptops would benefit greatly by double the amount of RAM. And then obviously, the AI need is gigantic. I don't see it going away. I mean, it may not grow as fast as now, but I don't see it growing away either. I get why the memory makers do not want to ba…
According to the recent article HBM memory is 3x less efficient wafer area wise than LPDDR; but the bandwidth is more than triple.
What if its in everyone's interest to buy computers at say 1/3rd the rate and switch everything over to HBM?
the discrepancy between compute and memory has been growing for ages, perhaps a painful switch to HBM is exactly what we need?
Would you rather have 3 intermediate computers with low memory bandwidth, or wait a little longer statistically so that we can all enjoy a new computer at 1/3rd the rate but much higher bandwidth than the area ratio?
Re: Memory has grown to nearly two-thirds of AI chip component costs
#106[deleted]
Re: Memory has grown to nearly two-thirds of AI chip component costs
#107Earlier quoted context omitted.
They will respond when people are loud enough. If memory stays at $1200 for 128GB for years and investigative journalists say it could be colluding, enough people will make enough noise. I’m sure Nvidia, Elon, Tim Cook, OpenAI, Anthropic are already whispering in Trump’s ears to do something.
> I’m sure Nvidia, Elon, Tim Cook, OpenAI, Anthropic are already whispering in Trump’s ears to do something You can't expect me to believe that any of those would want any kind of antitrust action against anybody.
Sure they do. They all have money interests in this. They all want lower memory prices.
Memory prices and shortages directly impact all of their profit margins and revenue.
Re: Memory has grown to nearly two-thirds of AI chip component costs
#108I assume that memory manufacturers don’t really care where the money is coming from, as long as the "numbers go up" game is working. NVIDIA in their recent quarterly report stopped categorizing "Geforce" as a single category, and merged it into "Edge-Computing". If you are a PC Gamer or PC Enthusiast as I am, then we have some dark times ahead.
Do we though? DLSS 5 changes that somewhat from a “we need powah” to “we need models”. I think the future consumer GPU market will be tuned for image and world inference while workstation cards will be tuned for image and video inference. The old way of thinking about this will come to an end when we stop looking at the render loop as the be-all-end-all… Or, we could be fucked.
If DLSS 5 becomes the norm it's possible that just makes things worse. The DLSS 5 demos required an entire separate card to run the model, though IIRC NVIDIA did claim it would eventually work on a single card. Given what the model is doing (yassifying the whole scene instead of just upscaling/reconstructing) it makes sense to me that it would increase compute demand instead of reduce it like previous versions of DLSS.
Re: Memory has grown to nearly two-thirds of AI chip component costs
#109Everything I read seems to suggest that RAM capacity is going to grow at 20-25% a year, which just doesn't seem good enough. Even in consumer use cases, phones and laptops would benefit greatly by double the amount of RAM. And then obviously, the AI need is gigantic. I don't see it going away. I mean, it may not grow as fast as now, but I don't see it growing away either. I get why the memory makers do not want to ba…
I mean the biggest risk is Chinese CXML benefits and capturing markets that others are leaving hanging and then being able to compete and push out the others when costs start to normalize. As for 20-25% growth not being enough, I think it's not that far off, if we assume data center build out plans hit a wall and slow down significantly, and the AI heat starts to cool off. I don't think 20-25% may be enough in the sh…
Is there any indication research is being focused on reducing menory footprint of inference for frontier class models? Is the low hanging fruit already gone there?
Re: Memory has grown to nearly two-thirds of AI chip component costs
#110I wonder why the hyperscalers aren't vertically integrating more and building their own fabs. Sure, a fab costs a billion dollars, but they're currently spending hundreds of billions of dollars purchasing chips from NVidia and others.
Because fabs are about the most complex cutting edge technology out there: the "rocket science" of our day (or one of them). And merely having the money is not sufficient. It would be very easy to blow several billion dollars and end up with nothing to show for it.
Just look at how Intel has struggled to compete in recent years, and they have been in the business for decades.