Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory has grown to nearly two-thirds of AI chip component costs

epoch.ai

Post reply on HN
Print view View original

91–100 of 524 posts

Re: Memory has grown to nearly two-thirds of AI chip component costs

#91

Post by vb-8448 »

johnvanommen wrote: post #56
Traubenfuchs wrote: post #46

Why did this happen so suddenly? Why were tech savy investors unable to figure this out when the datacenter craze had already started? How to explain this lag between quickly rising demand for all datacenter components besides memory?

Nine years after Google's seminal paper lit the fuse on AI, a total lack of manufacturing foresight has trapped over a trillion dollars of incoming capital in a hardware bottleneck. The entire sector is now facing a critical RAM starvation crisis where memory manufacturers are actively slow-rolling supply just to keep prices high and avoid running out entirely. This has created an unprecedented supply-and-demand dist…

Capex expenditure start exploding after covid with the chart going hockey stick at the end of 23/start of 24, almost 2.5 years ago.

A lot of capex is supposed to go into the datacentres, didn't they know that datacentres need to be filled among other stuff with RAM? I wonder if at some point we will discover that there is a shortage of fibre optic cables of SFPs ...

PS: Obviously armchair economist here too ... but for it doesn't seem too difficult to foresee the increase of the demand.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#92

Post by gruez »

johnvanommen wrote: post #69
layer8 wrote: post #67

Earlier quoted context omitted.

An unbelievably good deal at $4000 plus?

Possibly the best deal there is I really need to shut up, or bite the bullet and by one. If you graph the tokens per second on the 5090, your jaw will hit the floor at how cheap it is

With only 32gb of vram, you can only run small/quantized models, in which case what's the point? At $4000, that gets you 20 months of 10x claude or chagpt subscriptions, which provide far better models. You'd need some use case where you can tolerate worse models, and use a steady supply of them. That doesn't match most people's usage patterns.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#93

Post by bitwize »

dawnerd wrote: post #82
nik282000 wrote: post #52

Earlier quoted context omitted.

I feel like by the time the AI bubble bursts the PC market will be irreparably damaged. Manufactures who have been making "enterprise" parts aren't going to go back to making consumer parts because there will be no market for it. And with a glut of datacenters not making any money on slop, they are going to be repurposed for saas, stuff like OnShape but for every application. Most users don't seem to care about stori…

They’re going to pivot to you renting desktop cloud compute instead of owning anything.

Enjoy your HP laptop subscription, it's all the computer you're going to get moving forward.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#94

Post by kortilla »

dymk wrote: post #44
aurareturn wrote: post #39

Earlier quoted context omitted.

Bankruptcy risks. When Samsung had to sell memory at a loss after COVID, no one came to save them. They buffered their memory division using profits from their other businesses. That’s how Samsung survives memory downturns. According to some stories, this is how Samsung convinced TSMC to not enter the memory business - that you need a nation or other lines of business to prevent bankruptcies. The market has stabilize…

...And why do they go bankrupt? Because it's an incredibly capital intensive process, involving billions of dollars of investment into manufacturing infrastructure. That is to say, making memory is quite hard.

You’re confusing two independent things. There are simple processes that are extremely capital intensive with long lead times and then there are complex processes that require lots of R&D and industry secrets. Memory is the former in the chip world.

Other examples from outside of tech of easy but capital intensive processes are power generation and railroads. Very easy to do, but easy to end up broken by overbuilding for demand that fails to materialize or stay stable for the duration of your financing.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#95

Post by BigTTYGothGF »

aurareturn wrote: post #61
deadbabe wrote: post #45

Earlier quoted context omitted.

Current DOJ is corrupt as fuck, it will not happen. Get back to reality.

They will respond when people are loud enough. If memory stays at $1200 for 128GB for years and investigative journalists say it could be colluding, enough people will make enough noise. I’m sure Nvidia, Elon, Tim Cook, OpenAI, Anthropic are already whispering in Trump’s ears to do something.

> I’m sure Nvidia, Elon, Tim Cook, OpenAI, Anthropic are already whispering in Trump’s ears to do something

You can't expect me to believe that any of those would want any kind of antitrust action against anybody.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#96

Post by kortilla »

deadbabe wrote: post #45
aurareturn wrote: post #35

Earlier quoted context omitted.

They will. DOJ prosecuted memory makers in the late 90s and 2000s for collusion. This boom is magnitudes higher than before. The attention will be endless.

Current DOJ is corrupt as fuck, it will not happen. Get back to reality.

Corrupt doesn’t mean “acts without incentives”. If you assume a corrupt system, then the inputs are going to be who has influence over the DOJ. If there is more money to be made by breaking a cartel, then they would absolutely do it.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#97

Post by bcrosby95 »

JacobAsmuth wrote: post #58
mattmanser wrote: post #49

Earlier quoted context omitted.

It's gone up like 300% in cost in the last year.

Which surely is the highest it'll ever be! You're suggesting that the price will go down in the future? Would love to hear more about your thought process!

Are you saying we're entering a period where tech increases in price instead of decreases? I guess it depends upon time horizon, but your statement isn't very specific.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#99

Post by nik282000 »

bitwize wrote: post #93
dawnerd wrote: post #82

Earlier quoted context omitted.

They’re going to pivot to you renting desktop cloud compute instead of owning anything.

Enjoy your HP laptop subscription, it's all the computer you're going to get moving forward.

It's the reason I just build a new PC, despite the insane prices, I'd rather overpay than have reasonable prices but no stock to buy. With any luck I'll get 8-10 years out of this one and by then the PC landscape will be something else entirely.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#100

Post by gpm »

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).
Post reply on HN