AI PCs Aren't Good at AI: The CPU Beats the NPU • Hacker News Board

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#251

Post by shermantanktop » Thu, Oct 17, 2024, 2:40 PM UTC

AlexandrB wrote: post #16

isusmelj wrote: post #13

I think the results show that just in general the compute is not used well. That the CPU took 8.4ms and GPU took 3.2ms shows a very small gap. I'd expect more like 10x - 20x difference here. I'd assume that the onnxruntime might be the issue. I think some hardware vendors just release the compute units without shipping proper support yet. Let's see how fast that will change. Also, people often mistake the reason for…

> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. I have a sneaking suspicion that the real real reason for an NPU is marketing. "Oh look, NVDA is worth $3.3T - let's make sure we stick some AI stuff in our products too."

That’s how we got an explosion of interesting hardware in the early 80s - hardware companies attempting to entice consumers by claiming “blazing 16 bit speeds” or other nonsense. It was a marketing circus but it drove real investments and innovation over time. I’d hope the same could happen here.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#252

Post by HelloNurse » Thu, Oct 17, 2024, 2:42 PM UTC

hakfoo wrote: post #184

Spooky23 wrote: post #162

Earlier quoted context omitted.

Apple will have a completely AI capable product line in 18 months, with the major platforms basically done. Microsoft is built around the broken Intel tick/tick model of incremental improvement — they are stuck with OEM shitware that will take years to flush out of the channel. That means for AI, they are stuck with cloud based OpenAI, where NVIDIA has them by the balls and the hyperscalers are all fighting for GPU.…

We're still looking for "that is useful". The stuff they've been trying to sell AI to the public with is increasingly looking as absurd as every 1978 "you'll store your recipes on the home computer" argument. AI text became a Human Centipede story: Start with a coherent 10-word sentence, let AI balloon it into five pages of flowery nonsense, send it to someone else, who has their AI smash it back down to 10 meaningfu…

I expect this sort of thing to go out of fashion and/or be regulated after "AI" causes some large life loss, e.g. starting a war or designing a collapsing building.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#253

Post by shermantanktop » Thu, Oct 17, 2024, 2:44 PM UTC

mattnewton wrote: post #134

_zoltan_ wrote: post #93

Earlier quoted context omitted.

News flash: you're in the niche of the niche. People don't care about die size. I'd be willing to bet that the amount of money they are missing out on is miniscule and is by far offset by people's money who care about other stuff. Like you know, performance and battery life, just to stick to your examples.

That’s exactly what the poster is arguing- they are being sarcastic.

It whooshed over my head too. That’s the danger of sarcasm…it’s a cooperative form of humor but the other party might not get it.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#254

Post by VHRanger » Thu, Oct 17, 2024, 3:14 PM UTC

IshKebab wrote: post #53

Kon-Peki wrote: post #36

Earlier quoted context omitted.

Modern chips have to dedicate a certain percentage of the die to dark silicon [1] (or else they melt/throttle to uselessness), and these kinds of components count towards that amount. So the point of these components is to be used, but not to be used too much. Instead of an NPU, they could have used those transistors and die space for any number of things. But they wouldn't have put additional high performance CPU co…

If they aren't being used it would be better to dedicate the space to more SRAM.

SRAM is extremely hot, it's the very opposite of dark silicon

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#255

Post by hulitu » Thu, Oct 17, 2024, 3:30 PM UTC

isusmelj wrote: post #13

I think the results show that just in general the compute is not used well. That the CPU took 8.4ms and GPU took 3.2ms shows a very small gap. I'd expect more like 10x - 20x difference here. I'd assume that the onnxruntime might be the issue. I think some hardware vendors just release the compute units without shipping proper support yet. Let's see how fast that will change. Also, people often mistake the reason for…

> The whole point of the NPU is rather to focus on low power consumption

You know which chip has the lowest power consumption ? The one which is turned off. /s

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#256

Post by blharr » Thu, Oct 17, 2024, 3:34 PM UTC

noduerme wrote: post #232

kmeisthax wrote: post #21

Earlier quoted context omitted.

> I think some hardware vendors just release the compute units without shipping proper support yet This is Nvidia's moat. Everything has optimized kernels for CUDA, and maybe Apple Accelerate (which is the only way to touch the CPU matrix unit before M4, and the NPU at all). If you want to use anything else, either prepare to upstream patches in your ML framework of choice or prepare to write your own training and in…

I'm not sure why this is a moat. Isn't it just a matter of translation from CUDA to some other instruction set? If AMD or someone else makes cheaper hardware that does the same thing, it doesn't seem like a stretch for them to release a PyTorch patch or whatever.

Sure you can probably translate rough code and get something that "works" but all the thousands of small optimizations that are baked in are not trivial to just translate.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#257

Post by consteval » Thu, Oct 17, 2024, 3:42 PM UTC

JohnFen wrote: post #67

jsheard wrote: post #8

These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of the Snapdragon X which isolates the NPU specifically but AMDs equivalent with the same ~50 TOPS performance target can be seen here, and takes up about as much area as three high performance CPU cores: https://www.techpowerup.com/325035/amd-strix-point-silicon…

> These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. This has been my thinking. Today you have to go out of your way to buy a system with an NPU, so I don't have any. But tomorrow, will they just be included by default? That seems like a waste for those of us who aren't going to be running models. I wonder what other uses they could be put…

We already can't fit much more in CPUs. You can't just throw cores in there. CPUs these days are, like, 80% cache if you look at the die. We constantly shrink the compute part, but we don't put much more compute - that space is just used for cache.

So, I'm not sure that you're wasting much with the NPU. But I'm not an expert.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#258

Post by llm_nerd » Thu, Oct 17, 2024, 3:44 PM UTC

m00x wrote: post #96

NPUs are efficient, not especially fast. The CPU is much bigger than the NPU and has better cache access. Of course it'll perform better.

NPUs are actually incredibly fast for standard inference operations.

This benchmark is horribly flawed in many ways, and was so evidently useless that I'm surprised that they still decided to "publish" this. When your test gets 1% of the published performance, it's a good indication that things aren't being done correctly.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#259

Post by hedgehog » Thu, Oct 17, 2024, 3:45 PM UTC

touisteur wrote: post #235

bee_rider wrote: post #207

Earlier quoted context omitted.

I’m sure we’ll get GPNPU. Low precision matvecs could be fun to play with.

SHAVE from MOVIDIUS was fun, before Intel bought them out.

Did they become un-fun? There are a bunch on the new Intel CPUs.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#260

Post by consteval » Thu, Oct 17, 2024, 3:46 PM UTC

marginalia_nu wrote: post #240

CAP_NET_ADMIN wrote: post #90

Earlier quoted context omitted.

Beauty of CPUs - they'll chew through whatever bs code you throw at them at a reasonable speed.

I don't think this is correct. The difference between well optimized code and unoptimized code on the CPU is frequently at least an order of magnitude performance. Reason it doesn't seem that way is that the CPU is so fast we often bottleneck on I/O first. However, for compute-workloads like inference, it really does matter.

While this is true, the most effective optimizations you don't do yourself. The compiler or runtime does it. They get the low-hanging fruit. You can further optimize yourself, but unless your design is fundamentally bad, you're gonna be micro-optimizing.

gcc -O0 and -O2 has a HUGE performance gain. We don't really have anything to auto-magically do this for models, yet. Compilers are intimately familiar with x86.