Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory has grown to nearly two-thirds of AI chip component costs

epoch.ai

Post reply on HN
Print view View original

481–490 of 524 posts

Re: Memory has grown to nearly two-thirds of AI chip component costs

#481

Post by Curosinono »

zdragnar wrote: post #480
Curosinono wrote: post #475

Earlier quoted context omitted.

Thats not true though. First of, even per capita, the USA is at 8th place while China is 74. For sure China has a problem due to its gigantic size and amount of people to even be able to reach its people, but the health care costs are nowere as high as USA has. USA is actually the country with the highgest % of GDP spend for health care alone. Just checkout a YT video from an US American going to a normal chinese hos…

Rural China has a per capita GDP 1/10th of the US adjusting for purchasing power; if their health care wasn't cheaper only the very richest could afford anything at all. Even the wealthy coastal areas are 1/3 of the US. China and the US have the same life expectancy of 79 years, which is a very recent phenomenon due to the 2005-2018 changes I mentioned earlier. Obesity, lack of exercise and other cultural factors wei…

So your main argument is, that the US American health care system is good and has to be so expensive because people can afford to pay more?!

Feel free to us europe than or germany. We pay less and more people are better of than the US Americans.

US Americans pay 2.5 times more than the avg high income nation: https://www.commonwealthfund.org/international-health-policy...

Were is the benefit of paying that much more if you are not getting older than others?

Re: Memory has grown to nearly two-thirds of AI chip component costs

#482

Post by reactordev »

MrGilbert wrote: post #160
reactordev wrote: post #22

Earlier quoted context omitted.

Do we though? DLSS 5 changes that somewhat from a “we need powah” to “we need models”. I think the future consumer GPU market will be tuned for image and world inference while workstation cards will be tuned for image and video inference. The old way of thinking about this will come to an end when we stop looking at the render loop as the be-all-end-all… Or, we could be fucked.

From my point of view, I suppose we will enter a "Let AI generate entertainment" era. In which you just might rent everything, including games. No need for a beefy computer at home, you just need a slim endpoint: "Order yours now, for just $99.99 per month, hardware included! Order today, and you will get three months of 'Office Suite' for free, with a small additional cost of $49.99 after month 4. On a tight budget?…

On your Karna card…

Re: Memory has grown to nearly two-thirds of AI chip component costs

#483

Post by reactordev »

kg wrote: post #108
reactordev wrote: post #22

Earlier quoted context omitted.

Do we though? DLSS 5 changes that somewhat from a “we need powah” to “we need models”. I think the future consumer GPU market will be tuned for image and world inference while workstation cards will be tuned for image and video inference. The old way of thinking about this will come to an end when we stop looking at the render loop as the be-all-end-all… Or, we could be fucked.

If DLSS 5 becomes the norm it's possible that just makes things worse. The DLSS 5 demos required an entire separate card to run the model, though IIRC NVIDIA did claim it would eventually work on a single card. Given what the model is doing (yassifying the whole scene instead of just upscaling/reconstructing) it makes sense to me that it would increase compute demand instead of reduce it like previous versions of DLS…

The demos did, but look how far we have come in just two years? Running local LLMs, running local diffusion models, running local world models (albeit, barely a scene at this point). I do believe that in 10 years time, game will be producing latents and not events they way they do now. I also hope this means that VR can finally get the fidelity it needs to really take off.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#484

Post by zozbot234 »

trollbridge wrote: post #462
zozbot234 wrote: post #217

Earlier quoted context omitted.

You could set up swap space on Intel Optane media, it'll be about the same performance as DDR3 and sells for ~$1/GB on the secondary market. Though it will be a lot more power hungry than Flash, let alone DRAM - so not suitable for all uses.

Doesn’t that require an Optane capable system? Optane is a technology I’m still mad never became mainstream. It would be particularly useful today when trying to run local models.

Optane is available in NVMe form factor that will work basically everywhere. There's also Optane persistent DIMMs that only work in highly specific systems.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#485

Post by EnPissant »

zozbot234 wrote: post #437
EnPissant wrote: post #419

Earlier quoted context omitted.

>You don't need "very much" expert overlap to see aggregate gains at scale, you just need some of it I'm not sure what you are claiming. Decode is bottle-necked by memory bandwidth. To see a speed up of 2x, you have to ensure each expert weight memory fetch can be used by 2 parallel streams. What exactly is the average factor you are claiming for 5x parallel streams (due to "birthday paradox" factors)? The Birthday p…

An aggregate speedup of 2x is a lot, we don't need that in a local context. Local hardware is heavily constrained by power and thermals, not just bandwidth; so all we really care about is raising compute intensity for decode a little bit to relax the memory bandwidth constraint. The average factor will depend on just how sparse the model is and how far you can push parallelism, there isn't just one single answer.

But you won't see 2x expert re-use, the speedup with 5 streams will be tiny.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#486

Post by Tuna-Fish »

roenxi wrote: post #449
CGMthrowaway wrote: post #251

Earlier quoted context omitted.

You're making the point for him. Undersupply in a boom, store cash to ramp up capacity in a downturn. Presevres capital and avoids overcapacity during the turning

This sounds like a plan to sell less when prices are high and more when prices are low. That is one of the stupidest strategies a company could adopt. I assure you, the RAM makers are pumping out as much as they can and increasing capacity as fast as they think the market can handle. I'm not sure what world we live in when the scheming capitalists are all hunched around their table working out how to dodge selling th…

Building new capacity takes years. The idea is that the market is reliably cyclical, so you should expand when there is a downturn, when costs are low and you can afford the short-term capacity hits that expansion causes (fe. when you divide productive teams in two and fill both halves to full strength with new hires).

Re: Memory has grown to nearly two-thirds of AI chip component costs

#487

Post by BearOso »

eldenring wrote: post #193
BearOso wrote: post #152

Earlier quoted context omitted.

Unless there's a new paradigm, scaling up is all they can do to improve performance. They've shrunk down all the way to 1-bit models and all the low-hanging fruit is gone. There's no way for them to get much smaller, so they have to get bigger and faster to meet expectations.

this is just not true at all, there are massive leaps from algorithms, data, etc. every year. scale is one axis of many and you need to get them all correct.

What novel data hasn't already been used in training? What new algorithms are there? Can you post some links so we can read about them?

Re: Memory has grown to nearly two-thirds of AI chip component costs

#488

Post by zdragnar »

Curosinono wrote: post #481
zdragnar wrote: post #480

Earlier quoted context omitted.

Rural China has a per capita GDP 1/10th of the US adjusting for purchasing power; if their health care wasn't cheaper only the very richest could afford anything at all. Even the wealthy coastal areas are 1/3 of the US. China and the US have the same life expectancy of 79 years, which is a very recent phenomenon due to the 2005-2018 changes I mentioned earlier. Obesity, lack of exercise and other cultural factors wei…

So your main argument is, that the US American health care system is good and has to be so expensive because people can afford to pay more?! Feel free to us europe than or germany. We pay less and more people are better of than the US Americans. US Americans pay 2.5 times more than the avg high income nation: https://www.commonwealthfund.org/international-health-policy... Were is the benefit of paying that much more…

In a thread about China, I replied to a post about how American healthcare is bad.

My main argument, which is in my first comment, is that healthcare is a bad way to show the difference between China and the US, since they are actually have a lot of similarities, especially with access at the lower end of the income spectrum.

There's literally no reason to bring other countries into the conversation other than to say "US is bad", which does nothing to change the reality of healthcare access in China.

Re: Memory has grown to nearly two-thirds of AI chip component costs

#489

Post by therealdkz »

raincom wrote: post #476
spaceman_2020 wrote: post #386

Earlier quoted context omitted.

Indian bureaucracy thinks extremely short term. The state prioritizes extracting revenue over all else. So many baffling short-term decisions over everything from corporate tax breaks to a incentives for global events like Formula 1.l You really can’t expect the same bureaucratic setup to think in terms of the decade+ it would take to be competent at something like chips

Indian bureaucracy exists to enrich themselves for their next ten generations. The bureaucracy itself promotes, recruits incompetents. There is no way to reform such a bureaucracy. The whole system needs to be dismantled while an alternative system gets built; given the nature of Indian politics (freebies/jobs/reservations to certain groups; monthly stipends to certain groups by borrowing money while at the same time…

[dead]

Re: Memory has grown to nearly two-thirds of AI chip component costs

#490

Post by redanddead »

irishcoffee wrote: post #456
bitmasher9 wrote: post #365

Earlier quoted context omitted.

The future is here now, it’s just not evenly distributed. China will mass produce something to the point that it is widely distributed. That is how China acts as a great equalizer on a global scale. Another way China is a great equalizer is their willingness to do business with anyone that can pay.

China is willing to do business while giving zero fucks about the environment they are destroying and the global warming they are causing. It really blows my mind people support the china thing so much around here.

extremely hard to argue we give a fuck about the environment.

drop the bs

Post reply on HN