Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI PCs Aren't Good at AI: The CPU Beats the NPU

github.com

Post reply on HN
Print view View original

131–140 of 303 posts

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#131

Post by Dalewyn »

AlexandrB wrote: post #16
isusmelj wrote: post #13

I think the results show that just in general the compute is not used well. That the CPU took 8.4ms and GPU took 3.2ms shows a very small gap. I'd expect more like 10x - 20x difference here. I'd assume that the onnxruntime might be the issue. I think some hardware vendors just release the compute units without shipping proper support yet. Let's see how fast that will change. Also, people often mistake the reason for…

> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. I have a sneaking suspicion that the real real reason for an NPU is marketing. "Oh look, NVDA is worth $3.3T - let's make sure we stick some AI stuff in our products too."

There are no nerves in a neural processing unit, so yes: It's 300% bullshit marketing.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#132

Post by jcgrillo »

Dalewyn wrote: post #131
AlexandrB wrote: post #16

Earlier quoted context omitted.

> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. I have a sneaking suspicion that the real real reason for an NPU is marketing. "Oh look, NVDA is worth $3.3T - let's make sure we stick some AI stuff in our products too."

There are no nerves in a neural processing unit, so yes: It's 300% bullshit marketing.

Maybe the N secretly stands for NFT.. Like the tesla self driving hardware only smaller and made of silicon.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#133

Post by kalleboo »

ezst wrote: post #39
jsheard wrote: post #8

These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of the Snapdragon X which isolates the NPU specifically but AMDs equivalent with the same ~50 TOPS performance target can be seen here, and takes up about as much area as three high performance CPU cores: https://www.techpowerup.com/325035/amd-strix-point-silicon…

I can't wait for the LLM fad to be over so we get some sanity (and efficiency) back. I personally have no use for this extra hardware ("GenAI" doesn't help me in any way nor supports any work-related tasks). Worse, most people have no use for that (and recent surveys even show predominant hostility towards AI creep). We shouldn't be paying extra for that, it should be opt-in, and then it would become clear (by lookin…

> most people have no use for that

Apple originally added their NPUs before the current LLM wave to support things like indexing your photo library so that objects and people are searchable. These features are still very popular. I don't think these NPUs are fast enough for GenAI anyway.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#134

Post by mattnewton »

_zoltan_ wrote: post #93
renewiltord wrote: post #51

Earlier quoted context omitted.

I was telling someone this and they gave me link to a laptop with higher battery life and better performance than my own, but I kept explaining to them that the feature I cared most about was die size. They couldn't understand it so I just had to leave them alone. Non-technical people don't get it. Die size is what I care about. It's a critical feature and so many mainstream companies are missing out on my money beca…

News flash: you're in the niche of the niche. People don't care about die size. I'd be willing to bet that the amount of money they are missing out on is miniscule and is by far offset by people's money who care about other stuff. Like you know, performance and battery life, just to stick to your examples.

That’s exactly what the poster is arguing- they are being sarcastic.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#135

Post by gerdesj »

iml7 wrote: post #119
teilo wrote: post #111

Actual article title: Benchmarking Qualcomm's NPU on the Microsoft Surface Tablet Because this isn't about NPUs. It's about a specific NPU, on a specific benchmark, with a specific set of libraries and frameworks. So basically, this proves nothing.

But you can’t get more clicks. You have to attack enough people to get clicks.I feel like this place is becoming more and more filled with posts and titles like this.

Internet points are a bit crap but HN generally discusses things properly and off topic and downright weird stuff generally gets downvoted to doom.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#136

Post by cjbgkagh »

> We've tried to avoid that by making both the input matrices more square, so that tiling and reuse should be possible.

While it might be possible it would not surprise me if a number of possible optimizations had not made it into Onnx. It appears that Qualcomm does not give direct access to the NPU and users are expected to use frameworks to convert models over to it, and in my experience conversion tools generally suck and leave a lot of optimizations on the table. It could be less of NPUs suck and more of the conversions tools suck. I'll wait until I get direct access - I don't trust conversion tools.

My view of NPUs is that they're great for tiny ML models and very fast function approximations which is my intended use case. While LLMs are the new hotness there are huge number of specialized tasks that small models are really useful for.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#137

Post by gnabgib »

teilo wrote: post #111

Actual article title: Benchmarking Qualcomm's NPU on the Microsoft Surface Tablet Because this isn't about NPUs. It's about a specific NPU, on a specific benchmark, with a specific set of libraries and frameworks. So basically, this proves nothing.

The title is from the original article (https://petewarden.com/2024/10/16/ai-pcs-arent-very-good-at-...), the URL was changed by dang: https://news.ycombinator.com/item?id=41863591

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#138

Post by jcgrillo »

ezst wrote: post #39
jsheard wrote: post #8

These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of the Snapdragon X which isolates the NPU specifically but AMDs equivalent with the same ~50 TOPS performance target can be seen here, and takes up about as much area as three high performance CPU cores: https://www.techpowerup.com/325035/amd-strix-point-silicon…

I can't wait for the LLM fad to be over so we get some sanity (and efficiency) back. I personally have no use for this extra hardware ("GenAI" doesn't help me in any way nor supports any work-related tasks). Worse, most people have no use for that (and recent surveys even show predominant hostility towards AI creep). We shouldn't be paying extra for that, it should be opt-in, and then it would become clear (by lookin…

I just got an iphone and the whole photos thing is absolutely garbage. All I wanted to do was look through my damn photos and find one I took recently but it started playing some random music and organized them in no discernible order.. like it wasn't the reverse time sorted.. Idk what kind of fucked up "creative process" came up with that bullshit but I sure wish they'd unfuck it stat.

The camera is real good though.

Post reply on HN