Fairly misleading title, boiling down AI PCs to just the Microsoft Surface running Qualcomm
AI PCs Aren't Good at AI: The CPU Beats the NPU
121–130 of 303 posts
Re: AI PCs Aren't Good at AI: The CPU Beats the NPU
#122What exactly does Windows do with a NPU? I don't own an 'AI PC' but it seems like the NPUs are slow and can't run much. I know Apple's Neural Engine is used to power Face ID and the facial recognition stuff in Photos, among other things.
AI PC is just a marketing term, doesn't have any real substance
Yea, we know that. I believe that’s why the person you’re replying too was asking for examples of real usage.
Re: AI PCs Aren't Good at AI: The CPU Beats the NPU
#123The write up on the GitHub repo is much more informative than the blog. When running int8 matmul using onnx performance is ~0.6TF. https://github.com/usefulsensors/qc_npu_benchmark
Thanks—we changed the URL to that from https://petewarden.com/2024/10/16/ai-pcs-arent-very-good-at-... . Readers may way want to look at both, of course!
Oh, maybe also change the title? I flagged it because of the title/url not matching.
Re: AI PCs Aren't Good at AI: The CPU Beats the NPU
#124> the 45 trillion operations per second that’s listed in the specs Such a spec should be ideally be accompanied by code demonstrating or approximating the claimed performance. I can't imagine a sports car advertising a 0-100km/h spec of 2.0 seconds where a user is unable to get below 5 seconds.
I have some bad news for you regarding how car acceleration is measured.
Well, what is it?
Re: AI PCs Aren't Good at AI: The CPU Beats the NPU
#125Earlier quoted context omitted.
I think other ARM SoC vendors like Rockchip added NPUs before Apple, or at least around the same time.
I was curious so looked it up. Apple's first chip with an NPU was the A11 bionic in Sept 2017. Rockchip's was the RK1808 in Sept 2019.
Face ID was the first tent pole feature that ran on the NPU.
Re: AI PCs Aren't Good at AI: The CPU Beats the NPU
#126I think the results show that just in general the compute is not used well. That the CPU took 8.4ms and GPU took 3.2ms shows a very small gap. I'd expect more like 10x - 20x difference here. I'd assume that the onnxruntime might be the issue. I think some hardware vendors just release the compute units without shipping proper support yet. Let's see how fast that will change. Also, people often mistake the reason for…
They definitely aren't doing the timing properly, but also what you might think is timing is not what is generally marketed. But I will say, those marketed versions are often easier to compare. One such example is that if you're using GPU then have you actually considered that there's an asynchronous operation as part of your timing? If you're naively doing `time.time()` then what happens is this start = time.time()…
Important precision: the async part is absolutely not python specific, but comes from CUDA, indeed for performance, and you will have to use cuda events too in C++ to properly time it.
For ONNX the runtimes I know of are synchronous as we don't do each operation individually but whole models at once, there is no need for async, the timings should be correct.
Re: AI PCs Aren't Good at AI: The CPU Beats the NPU
#127These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of the Snapdragon X which isolates the NPU specifically but AMDs equivalent with the same ~50 TOPS performance target can be seen here, and takes up about as much area as three high performance CPU cores: https://www.techpowerup.com/325035/amd-strix-point-silicon…
> These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. This has been my thinking. Today you have to go out of your way to buy a system with an NPU, so I don't have any. But tomorrow, will they just be included by default? That seems like a waste for those of us who aren't going to be running models. I wonder what other uses they could be put…
Voice to text
Re: AI PCs Aren't Good at AI: The CPU Beats the NPU
#128These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of the Snapdragon X which isolates the NPU specifically but AMDs equivalent with the same ~50 TOPS performance target can be seen here, and takes up about as much area as three high performance CPU cores: https://www.techpowerup.com/325035/amd-strix-point-silicon…
Snapdragon X still has a full 12 cores (all same cores, it's homogeneous) and the Strix Point is also 12 cores but in a 4+8 configuration but with the "little" cores not sacrificing that much (nothing like the little cores in ARM's designs which might as well not even exist, they are a complete waste of silicon). Consumer software doesn't scale to that, so what are you going to do with more transistors allocated to the CPU?
It's not unlike why Apple puts so many video engines in their SoCs - they don't actually have much else to do with the transistor budget they can afford. Making single thread performance better isn't limited by transistor count anymore and software is bad at multithreading.
Re: AI PCs Aren't Good at AI: The CPU Beats the NPU
#129Earlier quoted context omitted.
I think other ARM SoC vendors like Rockchip added NPUs before Apple, or at least around the same time.
I was curious so looked it up. Apple's first chip with an NPU was the A11 bionic in Sept 2017. Rockchip's was the RK1808 in Sept 2019.
Google TPU was introduced around same time as apple. Basically everybody knew it can be something around that time, just don't know exactly how
Re: AI PCs Aren't Good at AI: The CPU Beats the NPU
#130Earlier quoted context omitted.
They definitely aren't doing the timing properly, but also what you might think is timing is not what is generally marketed. But I will say, those marketed versions are often easier to compare. One such example is that if you're using GPU then have you actually considered that there's an asynchronous operation as part of your timing? If you're naively doing `time.time()` then what happens is this start = time.time()…
Important precision: the async part is absolutely not python specific, but comes from CUDA, indeed for performance, and you will have to use cuda events too in C++ to properly time it. For ONNX the runtimes I know of are synchronous as we don't do each operation individually but whole models at once, there is no need for async, the timings should be correct.
Yes, it isn't python, it is... hardware. Not even CUDA specific. It is about memory moving around and optimization (remember, even the CPUs do speculative execution). I say a little more in the larger comment.
I'm less concerned about the CPU baseline and more concerned about the NPU timing. Especially given the other issues