Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory has grown to nearly two-thirds of AI chip component costs

epoch.ai

Post reply on HN
Print view View original

281–290 of 524 posts

Re: Memory has grown to nearly two-thirds of AI chip component costs

#281

Post by weitendorf »

gruez wrote: post #237
regularfry wrote: post #182

Earlier quoted context omitted.

If you can do what you need with qwen3.6-27b, it starts to look really interesting. That model is crazy good for the size, but it's a pain tweaking the params to run it on a 4090 with decent context and decent token speed. A 5090 looks tasty from that point of view, and only more so if you think in terms of the probability of that model being roflstomped by something in the same weight class in the next couple of yea…

>If you can do what you need with qwen3.6-27b, it starts to look really interesting. What's the use case here? Churning out massive amounts of slop code through autonomous agents? Running openclaw 24/7? I think the proliferation of codex and claude code, compared to any of the cheaper open models suggests that at least for most software development, the 50-75% discount of open models isn't worth the hassle of the dec…

I think there is a reasonable basis for taking a gamble that small models capable of fitting on a 32GB card will continue to advance over the next 5 years and eventually approach Gemini Flash 3.5 / Sonnet 4.6 levels of capabilities, which I would consider to be past the threshold of “probably worth the cost and hassle of running 24/7” if the upfront cost of the hardware was palatable.

My use case would primarily be in search, integration, and indexing other software projects with my own, as well as transcription/indexing of interesting video and audio content (eg Dwarkesh interviews) that I don’t have time to watch but want to easily search and apply to my projects, and search/indexing for useful information from things like Linux kernel and security mailing lists. Basically there is a lot of stuff that, if the cost were low enough, I would point a reasonably intelligent AI at to distill out useful information and apply it to my projects, or just cherry pick the interesting things out and surface them to me so I don’t have to wade through all the mundane stuff and man-made slop getting in the way.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#282

Post by dominotw »

brandensilva wrote: post #183
radialstub wrote: post #161

Earlier quoted context omitted.

The memory makers will not expand demand drastically. It is in the nature of their business to keep the market under-supplied, otherwise the following oversupply will kill them. Instead, supply is just rerouted from less profitable segments such as mobile and personal computing.

China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply. But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer. I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.

man i keep thinking. why cant india get into stuff like this. Do their own manhattan project to build factories and tech for this and immigrate experts with high salaries.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#284

Post by epolanski »

dylan604 wrote: post #262
esseph wrote: post #257

Earlier quoted context omitted.

This has nothing to do about nationality, it has everything to do with building and running a brand new, highly technical, mass production facility.

And historical record of the lack of QA coming from Chinese manufacturing

China is a gigantic country where one in 6 humans live that either produce directly or indirectly, 70%+ of the world's goods.

It's quite difficult to make general statements at such a gargantuan scale encompassing every single sector.

China has an abundance of terrific QA in electronics and advanced technologies as much as it has an abundance of the opposite, just simply due to its sheer size.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#285

Post by epolanski »

jmalicki wrote: post #168
Waterluvian wrote: post #121

Earlier quoted context omitted.

What’s the lifespan/refurbishability of the capex elements like the “GPU” modules or even the DRAM soldered into them?

For lifespan, AWS is still running a ton of T4 GPUs from 2018, that power a lot of computer vision models. A ton of these will have a long life, not all ML is about frontier LLMs.

How can it be economically viable to still run them?

You can get 100x the output with the same energy use.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#286

Post by throawayonthe »

xbmcuser wrote: post #261
overfeed wrote: post #216

Earlier quoted context omitted.

> SK Hynix and Samsung are South Korean. The Korean memory makers are playing the same game as Micron and simply moving existing capacity up-market. GP was referring to upstart Chinese memory manufacturers like ChangXin, who - if their yields manage to catch the wave - could not have asked for a more favorable market after the big 3 have abandoned the consumer segment. Consumers who would have otherwise turned up the…

Chinese manufacturers will probably takeover consumer ram that most of us use as current manufacturing contracts expire and Samsung SK and micron move all their production to HBM for data centers. Corsair recently released chinese chips based DDR5 sticks.

> Corsair recently released chinese chips based DDR5 sticks.

hm interesting

https://www.tomshardware.com/pc-components/ddr5/chinese-memo...

Re: Memory has grown to nearly two-thirds of AI chip component costs

#287

Post by thfuran »

zozbot234 wrote: post #225
thfuran wrote: post #138

Earlier quoted context omitted.

Not many workloads are RAM bandwidth limited. Power and latency are much more common bottlenecks, and HBM loses on both of those.

Multicore workloads do tend to hit RAM bandwidth limits before they hit power constraints. If you do the math, running at max frequency and core utilization would usually imply you could only access a byte or so per core clock cycle. Perhaps a mere handful of bytes for the highest-performance systems with in-package RAM.

What percent of the time do you think the average consumer computing device spends fully clocked up, let alone fully saturated on every core?

Re: Memory has grown to nearly two-thirds of AI chip component costs

#288

Post by daringrain32781 »

KronisLV wrote: post #20

I'm not moving past my DDR4 build (and the 32 GB of DDR4 2133 MHz backup chips I still have around from way back, before I got the current 3200 MHz ones) until the prices go back to being at least partially sane. This also means that CPU manufacturers are not getting my money (since the 5800X is fine for now) and I have no reason to get a new GPU either (though admittedly the B580 isn't perfect).

[dead]

Re: Memory has grown to nearly two-thirds of AI chip component costs

#289

Post by com2kid »

minraws wrote: post #129
galangalalgol wrote: post #109

Earlier quoted context omitted.

Is there any indication research is being focused on reducing menory footprint of inference for frontier class models? Is the low hanging fruit already gone there?

Low hanging? how low hanging are we talking, the basic stuff is gone. Largely big challenges around quantization were solved 2 years ago, and we have just been improving from there. But can massive gains still be made? Definitely. The entire AI hype is based on the paper Attention is all you need, and Attention is basically loading a huge matrix of all the tokens in memory, how well you can optimize this attention la…

The other side of this is how powerful small and medium parameter models are.

24b param models today are way more powerful than 24b param models 2 years ago.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#290

Post by gruez »

weitendorf wrote: post #281
gruez wrote: post #237

Earlier quoted context omitted.

>If you can do what you need with qwen3.6-27b, it starts to look really interesting. What's the use case here? Churning out massive amounts of slop code through autonomous agents? Running openclaw 24/7? I think the proliferation of codex and claude code, compared to any of the cheaper open models suggests that at least for most software development, the 50-75% discount of open models isn't worth the hassle of the dec…

I think there is a reasonable basis for taking a gamble that small models capable of fitting on a 32GB card will continue to advance over the next 5 years and eventually approach Gemini Flash 3.5 / Sonnet 4.6 levels of capabilities, which I would consider to be past the threshold of “probably worth the cost and hassle of running 24/7” if the upfront cost of the hardware was palatable. My use case would primarily be i…

>My use case would primarily be in search, integration, and indexing other software projects with my own, as well as transcription/indexing of interesting video and audio content (eg Dwarkesh interviews) that I don’t have time to watch but want to easily search and apply to my projects, and search/indexing for useful information from things like Linux kernel and security mailing lists. Basically there is a lot of stuff that, if the cost were low enough, I would point a reasonably intelligent AI at to distill out useful information and apply it to my projects, or just cherry pick the interesting things out and surface them to me so I don’t have to wade through all the mundane stuff and man-made slop getting in the way.

All of that feels like something that a $20 chatgpt pro subscription is for, maybe with slightly better tool use capabilities. There's no way that a $4000 purchase on a GPU would ever be worth it if all you're doing is running a handful of queries per day.

Post reply on HN