Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory has grown to nearly two-thirds of AI chip component costs

epoch.ai

Post reply on HN
Print view View original

151–160 of 524 posts

Re: Memory has grown to nearly two-thirds of AI chip component costs

#151

Post by andrepd »

gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

I wonder if we will see an adoption of alternative floating point formats. IEEE floats are notoriously terrible at lower widths (<= 16 bits). Floating point formats such as posits do much better at 16 or 8 bits. If you could train at 16 bits per value instead of 32, and suffer a much smaller inaccuracy penalty than you would from IEEE32 to IEEE16...

Re: Memory has grown to nearly two-thirds of AI chip component costs

#152

Post by BearOso »

eldenring wrote: post #143
gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

2-3x is completely dwarfed by the remaining improvements in training which is still in its infancy relatively

Unless there's a new paradigm, scaling up is all they can do to improve performance. They've shrunk down all the way to 1-bit models and all the low-hanging fruit is gone. There's no way for them to get much smaller, so they have to get bigger and faster to meet expectations.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#153

Post by gpm »

eldenring wrote: post #143
gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

2-3x is completely dwarfed by the remaining improvements in training which is still in its infancy relatively

Probably, but at some point we're very likely to run out of significant training improvements and it's not clear that we'll see that point coming from a long way out.

Likewise it's probably dwarfed by improvements in how we make dram - continuing the roughly exponential (maybe a bit less recently) scaling of chips - but not necessarily.

The 2x from returning to previous costs is interesting because it's practically guaranteed, and it's on top of everything else. We're just currently "overpaying" (relative to the stable market price) for the manufacture of dram because of a sudden increase in demand.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#155

Post by shevy-java »

gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

> a path to a ~3x hardware cost reduction

Really?

How long do we have to wait until that ... cost reduction hits us?

Re: Memory has grown to nearly two-thirds of AI chip component costs

#156

Post by Npovview »

Legend2440 wrote: post #12
positron26 wrote: post #3

The algorithm advances are going to crash this so hard.

Or will more efficient algorithms just mean we run even more AI models, increasing the demand for AI chips even more?

I heard Greg Brockman on a podcast saying they are limited by computer and memory. They have line of sight in solving many different kinds of problems. But they also have to survive in the meantime. Hence the focus on enterprise recently. They could just ask Government to fund them doing other research areas

Re: Memory has grown to nearly two-thirds of AI chip component costs

#157

Post by dawnerd »

Forgeties79 wrote: post #144
dawnerd wrote: post #80

Earlier quoted context omitted.

I’m so mad I didn’t max out my main server when I had the chance. Used enterprise sticks were dirt cheap on eBay.

Used enterprise HDD’s also jacked up now. It’s absurd lol

Yep mad about that too. I was about half way through upgrading my 45 drives server when they started to go up.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#158

Post by shevy-java »

slicktux wrote: post #2

I bought 96GB of RAM a couple of years ago for ~$250. That same RAM now costs $1200!

My main computer has 64GB. I bought that one in late 2022 or so.

Looking at the current prices, even of the same RAM, is just insane. Those companies really need to pay us compensation damage here. The whole "free market" notion does not work when you have de-facto monopolies and mega-corporations abuse average Joe and average Jane.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#159

Post by throwatdem12311 »

aunty_helen wrote: post #66
oceansky wrote: post #6

Awful time for gamers and PC hobbyists not fully into AI.

This is 100% going to kill the home built pc market. When I started building gaming pcs, the top top card was 750$ (NZD). Now they’re 10,000 just for the gpu and another 1-2000 for ram. People used to get into gaming pcs as an affordable hobby, now it’s making general aviation look like plan B.

Don’t you worry - Microsoft and Amazon will have you covered with cloud streaming.

Can’t afford a computer because they bought up all the supply? They’ll conveniently sell it back to you with a subscription!

You’ll own nothing and be happy.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#160

Post by MrGilbert »

reactordev wrote: post #22
MrGilbert wrote: post #17

I assume that memory manufacturers don’t really care where the money is coming from, as long as the "numbers go up" game is working. NVIDIA in their recent quarterly report stopped categorizing "Geforce" as a single category, and merged it into "Edge-Computing". If you are a PC Gamer or PC Enthusiast as I am, then we have some dark times ahead.

Do we though? DLSS 5 changes that somewhat from a “we need powah” to “we need models”. I think the future consumer GPU market will be tuned for image and world inference while workstation cards will be tuned for image and video inference. The old way of thinking about this will come to an end when we stop looking at the render loop as the be-all-end-all… Or, we could be fucked.

From my point of view, I suppose we will enter a "Let AI generate entertainment" era. In which you just might rent everything, including games. No need for a beefy computer at home, you just need a slim endpoint:

"Order yours now, for just $99.99 per month, hardware included! Order today, and you will get three months of 'Office Suite' for free, with a small additional cost of $49.99 after month 4. On a tight budget? Switch to the yearly subscription, and pay comfortably in 18 installments."

Post reply on HN