Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory has grown to nearly two-thirds of AI chip component costs

epoch.ai

Post reply on HN
Print view View original

361–370 of 524 posts

Re: Memory has grown to nearly two-thirds of AI chip component costs

#361

Post by Dylan16807 »

epolanski wrote: post #285
jmalicki wrote: post #168

Earlier quoted context omitted.

For lifespan, AWS is still running a ton of T4 GPUs from 2018, that power a lot of computer vision models. A ton of these will have a long life, not all ML is about frontier LLMs.

How can it be economically viable to still run them? You can get 100x the output with the same energy use.

Going by the stats on wikipedia, T4 and B300 both do about one teraflop of half-precision math per watt? Where are the efficiency gains?

Edit: It looks like they replaced INT8 and INT4 with FP8 and FP4, with the same speedups of 2x and 4x relative to FP16. That's an improvement but not that big of an improvement.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#362

Post by daemin »

gpm wrote: post #352
raincole wrote: post #341

Earlier quoted context omitted.

Not just DRAM market, but the GPU market soon. China is the great equalizer of the world.

> Not just DRAM market, but the GPU market soon China doesn't have EUV fabs... They've pushed DUV impressively far... but until they get EUV working industrially (and reasonable timelines are at least 2-4 years for that) it shouldn't be possible for them to compete for that market. > China is the great equalizer of the world. China is hardly an egalitarian society...

Not an egalitarian society, but their companies have a honey-badger like mentality from what I have read, where they ruthlessly reduce costs and margin down past where non-Chinese companies cannot compete.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#363

Post by noosphr »

brandensilva wrote: post #183
radialstub wrote: post #161

Earlier quoted context omitted.

The memory makers will not expand demand drastically. It is in the nature of their business to keep the market under-supplied, otherwise the following oversupply will kill them. Instead, supply is just rerouted from less profitable segments such as mobile and personal computing.

China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply. But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer. I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.

Help us Xi Jinping, you're our only hope.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#364

Post by da_chicken »

shevy-java wrote: post #155
gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

> a path to a ~3x hardware cost reduction Really? How long do we have to wait until that ... cost reduction hits us?

All the projections I've seen have said that the earliest we might see the curve flatten is 2030.

It just takes that long to get a fab up and running.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#365

Post by bitmasher9 »

gpm wrote: post #352
raincole wrote: post #341

Earlier quoted context omitted.

Not just DRAM market, but the GPU market soon. China is the great equalizer of the world.

> Not just DRAM market, but the GPU market soon China doesn't have EUV fabs... They've pushed DUV impressively far... but until they get EUV working industrially (and reasonable timelines are at least 2-4 years for that) it shouldn't be possible for them to compete for that market. > China is the great equalizer of the world. China is hardly an egalitarian society...

The future is here now, it’s just not evenly distributed. China will mass produce something to the point that it is widely distributed. That is how China acts as a great equalizer on a global scale.

Another way China is a great equalizer is their willingness to do business with anyone that can pay.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#367

Post by raincom »

dominotw wrote: post #282
brandensilva wrote: post #183

Earlier quoted context omitted.

China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply. But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer. I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.

man i keep thinking. why cant india get into stuff like this. Do their own manhattan project to build factories and tech for this and immigrate experts with high salaries.

Thanks to Indian constitution and other laws, incompetence is heavily praised and promoted. Go to any public high/elementary schools, go to any private colleges, it is rampant. It has taken three generations to produce these bad results. Of course, I am not saying that students are dumb. It is just that many smart kids in villages if given an opportunity, will leave India in a heart beat because they don't see a bright future for their future kids.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#369

Post by alexwwang »

noosphr wrote: post #363
brandensilva wrote: post #183

Earlier quoted context omitted.

China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply. But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer. I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.

Help us Xi Jinping, you're our only hope.

It’s a horrible thought. Really horrible. You should come to China and work in those factories and mines for some years by yourself.
Post reply on HN