Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The RAM shortage comes for us all

jeffgeerling.com

Post reply on HN
Print view View original

111–120 of 416 posts

Re: The RAM shortage comes for us all

#111

Post by hathawsh »

The article suggests that because the power and cooling are customized, it would take a ton of effort to run the new AI servers in a home environment, but I'm skeptical of that. Home-level power and cooling are not difficult these days. I think when the next generation of AI hardware comes out (in 3-5 years), there will be a large supply of used AI hardware that we'll probably be able to repurpose. Maybe we'll sell them as parts. It won't be plug-and-play at first, but companies will spring up to figure it out.

If not, what would these AI companies do with the huge supply of hardware they're going to want to get rid of? I think a secondary market is sure to appear.

Re: The RAM shortage comes for us all

#114

Post by Thegn »

jl6 wrote: post #97

Can someone explain why OpenAI is buying DDR5 RAM specifically? I thought LLMs typically ran on GPUs with specialised VRAM, not on main system memory. Have they figured out how to scale using regular RAM?

They didn't buy DDR5 - they bought raw wafer capacity and a ton of it at that.

Re: The RAM shortage comes for us all

#115

Post by MPSimmons »

fullstop wrote: post #101
MPSimmons wrote: post #89

This reminds me of the recent LaurieWired video presenting a hypothetical of, "what if we stopped making CPUs": https://www.youtube.com/watch?v=L2OJFqs8bUk Spoiler, but the answer is basically that old hardware rules the day because it lasts longer and is more reliable of timespans of decades. DDR5 32GB is currently going for ~$330 on Amazon DDR4 32GB is currently going for ~$130 on Amazon DDR3 32GB is currently goin…

If everyone went for DDR4 and DDR3, surely the cost would go up. There is no additional supply there, as they are no longer being made.

Undoubtably the cost would go up, but nobody is building out datacenters full of DDR4, either, so I don't figure it would go up nearly as much as DDR5 is right now.

Re: The RAM shortage comes for us all

#118

Post by acdha »

fullstop wrote: post #101
MPSimmons wrote: post #89

This reminds me of the recent LaurieWired video presenting a hypothetical of, "what if we stopped making CPUs": https://www.youtube.com/watch?v=L2OJFqs8bUk Spoiler, but the answer is basically that old hardware rules the day because it lasts longer and is more reliable of timespans of decades. DDR5 32GB is currently going for ~$330 on Amazon DDR4 32GB is currently going for ~$130 on Amazon DDR3 32GB is currently goin…

If everyone went for DDR4 and DDR3, surely the cost would go up. There is no additional supply there, as they are no longer being made.

At some point that’s true, but don’t they run the n-1 or 2 generation production lines for years after the next generation launches? There’s a significant capital investment there and my understanding is that the cost goes down significantly over the lifetime as they dial in the process so even though the price is lower it’s still profitable.

Re: The RAM shortage comes for us all

#119

Post by fullstop »

jl6 wrote: post #108
fullstop wrote: post #103

Earlier quoted context omitted.

They're not. They are buying wafers / production capacity to make HBM so there is less DDR5 supply.

OK, fair enough, but what are OpenAI doing buying production capacity rather than, say, paying NVIDIA to do it? OpenAI aren’t the ones making the hardware?

Because they can provide the materials to NVIDIA for production and prevent Google, Anthropic, etc from having them.

Re: The RAM shortage comes for us all

#120

Post by SimianSci »

roadside_picnic wrote: post #37

I know this is mostly paranoid thinking on my behalf, but it almost feels like this is a conscious effort to attempt to destroy "personal" computing. I've been a huge advocate for local, open, generative AI as the best resistance to massive take-over by large corporations controlling all of this content creation. But even as it is (or "was" I should say), running decent models at home is prohibitively expensive for m…

Wouldn't the easy answer to this be increased efficiency of RAM usage?

RAM being plentiful and cheap led to a lot of software development being very RAM-unaware, allowing the inefficiencies of programs to be mostly obfuscated from the user. If RAM prices continue rising, the semi-apocalytic consumer fiction you've spun here would require that developers not change their behaviors when it comes to software they write. There will be an equillibrium in the market that still allows the entry of consumer PC's it will just mean devices people buy will have less available RAM than is typical. The demand will eventually match up to the change in supply as is typical of supply/demand issues and not continuously rise into an infinite horizon.

Post reply on HN