Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory has grown to nearly two-thirds of AI chip component costs

epoch.ai

Post reply on HN
Print view View original

391–400 of 524 posts

Re: Memory has grown to nearly two-thirds of AI chip component costs

#391

Post by Closi »

XorNot wrote: post #256
Closi wrote: post #238

Earlier quoted context omitted.

Although on the flipside, let's pretend it's 2017's and you are Nvidia selling GPU's for Bitcoin - maybe demand will dry up at some point? Do you stop scaling production as this might be the max of the market, or do you follow the market and increase production? It's always easier to see the right move in hindsight!

Its a lot easier to commit to spending billions of dollars in a hypothetical then reality.

But it's also easier if you have a market cap of 2 trillion and are worried about your competitors scaling to meet demand.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#392

Post by Closi »

fragmede wrote: post #258
Closi wrote: post #238

Earlier quoted context omitted.

Although on the flipside, let's pretend it's 2017's and you are Nvidia selling GPU's for Bitcoin - maybe demand will dry up at some point? Do you stop scaling production as this might be the max of the market, or do you follow the market and increase production? It's always easier to see the right move in hindsight!

Nvidia doesn't own fabs though, TSMC does. By 2017, ASICs for Bitcoin were well underway. Ethereum hadn't switched to PoS, and wouldn't until 2022. For that specific question, the answer is yes, because the GTX 1080 Ti is/was a monster card, and the crypto miners have a somewhat predictable demand for them, so there's some modeling you can do based on demand for the 2016 generation of cards. The question is ofc, if y…

Nvidia doesn't own fabs, but TSMC is doing a massive global fab expansion.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#394

Post by viccis »

alexwwang wrote: post #369
noosphr wrote: post #363

Earlier quoted context omitted.

Help us Xi Jinping, you're our only hope.

It’s a horrible thought. Really horrible. You should come to China and work in those factories and mines for some years by yourself.

Why would I, as an American engineer and user of tech hardware from China for quite some time now, need to immerse myself in the Chinese factories as if they are somehow worse than other ones throughout the world?

Thanks, please give my regards to Kash Patel.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#395

Post by oblio »

weitendorf wrote: post #270
gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

If you factor in Nvidia’s profit margin due to the scarcity of the current bleeding-edge chips there is a path to a much larger cost reduction still. There’s a lot to criticize Sam Altman for saying or popularizing culturally but I’ve come to think his “this is the worst it will ever be” is, in the long run, actually a very intriguing and underrated point. In a decade training LLMs to the current level of sophisticat…

These crazy hardware price increases will probably delay everything by at least 2-5 years. Then add at least 5-10 years for all these refinements and optimizations to permeate universally.

Until everything matures, most likely the current iteration of OpenAI and Anthropic will be long gone, along with their current business models.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#396

Post by makingstuffs »

gpm wrote: post #352
raincole wrote: post #341

Earlier quoted context omitted.

Not just DRAM market, but the GPU market soon. China is the great equalizer of the world.

> Not just DRAM market, but the GPU market soon China doesn't have EUV fabs... They've pushed DUV impressively far... but until they get EUV working industrially (and reasonable timelines are at least 2-4 years for that) it shouldn't be possible for them to compete for that market. > China is the great equalizer of the world. China is hardly an egalitarian society...

There are no egalitarian societies. Societies in the west favour the super rich and believing anything else is simply delusional. Sorry to burst the bubble.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#397

Post by oblio »

intelkishan wrote: post #353
BearOso wrote: post #152

Earlier quoted context omitted.

Unless there's a new paradigm, scaling up is all they can do to improve performance. They've shrunk down all the way to 1-bit models and all the low-hanging fruit is gone. There's no way for them to get much smaller, so they have to get bigger and faster to meet expectations.

This hasn’t been true for the past 2 years

Is this based on an assumption that Opus 4.7 & co are equivalent or smaller to Opus 4.5 & co? I highly doubt the advanced models (Opus, Pro, etc) aren't biggen than the standard ones (Sonnet, Flash, etc) and fairly sure newer models are bigger than older ones.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#398

Post by liccil »

gpm wrote: post #100

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

What demand? Can't shake the notion that it's fictive considering the amount od data centers being built and GPUs sitting in containers, where they will spend quite some time before being even integrated, even more until used...

Re: Memory has grown to nearly two-thirds of AI chip component costs

#399

Post by esperent »

shdh wrote: post #201
brandensilva wrote: post #183

Earlier quoted context omitted.

China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply. But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer. I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.

Yeah, more global competition in DRAM would be great. SK Hynix and Samsung are South Korean.

Right, so exactly two countries in the world control most of the memory market (US and Korra). As the person above said, more global competition would be great.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#400

Post by EnPissant »

zozbot234 wrote: post #233
Galanwe wrote: post #140

Earlier quoted context omitted.

The 5090 is crap for inference. Unless you like dummy models, sure they will run at light speed. All the rage is MoE with 500B-1T weights nowadays.

MoE is fine. You can put the shared weights on the 5090 (will fit handily even for the largest models) and expert weights on CPU, possibly with weights offload from storage.

Even if you could fit a 500B model's expert weights in very fast system RAM, it would run so slow as to be useless.
Post reply on HN