Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI PCs Aren't Good at AI: The CPU Beats the NPU

github.com

Post reply on HN
Print view View original

151–160 of 303 posts

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#151

Post by grugagag »

kalleboo wrote: post #133
ezst wrote: post #39

Earlier quoted context omitted.

I can't wait for the LLM fad to be over so we get some sanity (and efficiency) back. I personally have no use for this extra hardware ("GenAI" doesn't help me in any way nor supports any work-related tasks). Worse, most people have no use for that (and recent surveys even show predominant hostility towards AI creep). We shouldn't be paying extra for that, it should be opt-in, and then it would become clear (by lookin…

> most people have no use for that Apple originally added their NPUs before the current LLM wave to support things like indexing your photo library so that objects and people are searchable. These features are still very popular. I don't think these NPUs are fast enough for GenAI anyway.

I wish I could turn that off on my phone.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#152

Post by wmf »

kllrnohj wrote: post #128
jsheard wrote: post #8

These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of the Snapdragon X which isolates the NPU specifically but AMDs equivalent with the same ~50 TOPS performance target can be seen here, and takes up about as much area as three high performance CPU cores: https://www.techpowerup.com/325035/amd-strix-point-silicon…

Snapdragon X still has a full 12 cores (all same cores, it's homogeneous) and the Strix Point is also 12 cores but in a 4+8 configuration but with the "little" cores not sacrificing that much (nothing like the little cores in ARM's designs which might as well not even exist, they are a complete waste of silicon). Consumer software doesn't scale to that, so what are you going to do with more transistors allocated to t…

GPU "infinity" cache would increase 3D performance and there's a rumor that AMD removed it to make room for the NPU. They're not out of ideas for features to put on the chip.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#153

Post by MBCook »

brookst wrote: post #145
itishappy wrote: post #23

Earlier quoted context omitted.

I assume you're both right. I'm sure NPUs exist to fill a very real niche, but I'm also sure they're being shoehorned in everywhere regardless of product fit because "AI big right now."

The shoehorning only works if there is buyer demand. As a company, if customers are willing to pay a premium for a NPU, or if they are unwilling to buy a product without one, it is not your place to say “hey we don’t really believe in the AI hype so we’re going to sell products people don’t want to prove a point”

Is there demand? Or do they just assume there is?

If they shove it in every single product and that’s all anyone advertises, whether consumers know it will help them or not, you don’t get a lot of choice.

If you want the latest chip, you’re getting AI stuff. That’s all there is to it.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#154

Post by AlotOfReading »

jcgrillo wrote: post #147
Kon-Peki wrote: post #36

Earlier quoted context omitted.

Modern chips have to dedicate a certain percentage of the die to dark silicon [1] (or else they melt/throttle to uselessness), and these kinds of components count towards that amount. So the point of these components is to be used, but not to be used too much. Instead of an NPU, they could have used those transistors and die space for any number of things. But they wouldn't have put additional high performance CPU co…

Question--what's to be lost by making your features sufficiently not dense to allow them to cool at full tilt?

Messes with timing, among other things. A lot of those structures are relatively fixed blocks that are designed for specific sizes. Signals take more time to propagate longer distances, and longer conductors have worse properties. Dense and hot is faster and more broadly useful.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#155

Post by -mlv »

brookst wrote: post #146
Dalewyn wrote: post #131

Earlier quoted context omitted.

There are no nerves in a neural processing unit, so yes: It's 300% bullshit marketing.

Neural is an adjective. Adjectives do not require their associated nouns to be present. See also: digital computers have mo fingers at all.

I always thought 'digital' referred to numbers, not fingers.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#156

Post by bdd8f1df777b »

brookst wrote: post #145
itishappy wrote: post #23

Earlier quoted context omitted.

I assume you're both right. I'm sure NPUs exist to fill a very real niche, but I'm also sure they're being shoehorned in everywhere regardless of product fit because "AI big right now."

The shoehorning only works if there is buyer demand. As a company, if customers are willing to pay a premium for a NPU, or if they are unwilling to buy a product without one, it is not your place to say “hey we don’t really believe in the AI hype so we’re going to sell products people don’t want to prove a point”

There are two kinds of buyer demands: product, buyers, and the stock buyers. The AI hype can certainly convince some of the stock buyers.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#157

Post by bdd8f1df777b »

rjsw wrote: post #91
kmeisthax wrote: post #22

Earlier quoted context omitted.

You forget "Because Apple is doing it", too.

I think other ARM SoC vendors like Rockchip added NPUs before Apple, or at least around the same time.

Even if it were true, they wouldn’t have the same influence as Apple has.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#158

Post by bdd8f1df777b »

-mlv wrote: post #155
brookst wrote: post #146

Earlier quoted context omitted.

Neural is an adjective. Adjectives do not require their associated nouns to be present. See also: digital computers have mo fingers at all.

I always thought 'digital' referred to numbers, not fingers.

The derivative meaning has been use so widely that it has surpassed its original one in usage. But it doesn’t change the fact that it originally refers to the fingers.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#159

Post by Spooky23 »

AlexandrB wrote: post #16
isusmelj wrote: post #13

I think the results show that just in general the compute is not used well. That the CPU took 8.4ms and GPU took 3.2ms shows a very small gap. I'd expect more like 10x - 20x difference here. I'd assume that the onnxruntime might be the issue. I think some hardware vendors just release the compute units without shipping proper support yet. Let's see how fast that will change. Also, people often mistake the reason for…

> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. I have a sneaking suspicion that the real real reason for an NPU is marketing. "Oh look, NVDA is worth $3.3T - let's make sure we stick some AI stuff in our products too."

Microsoft needs to throw something in the gap to slow down MacBook attrition.

The M processors changed the game. My teams support 250k users. I went from 50 MacBooks in 2020 to over 10,000 today. I added zero staff - we manage them like iPhones.

Post reply on HN