Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI PCs Aren't Good at AI: The CPU Beats the NPU

github.com

Post reply on HN
Print view View original

31–40 of 303 posts

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#31

Post by refulgentis »

eightysixfour wrote: post #27
nickpsecurity wrote: post #25

Earlier quoted context omitted.

To take half as long, doesn’t it have to perform twice as fast? Or am I misreading your comment?

No, you can have latency that is independent of compute performance. The CPU/GPU may have other tasks and the work has to wait for the existing threads to finish, or for them to clock up, or have slower memory paths, etc. If you and I have the same calculator but I'm working on a set of problems and you're not, and we're both asked to do some math, it may take me longer to return it, even though the instantaneous per…

In isolation, makes sense.

Wouldn't it be odd for OP to present examples that are the opposite of their claim, just to get us thinking about "well the CPU is busy?"

Curious for their input.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#32

Post by refulgentis »

eightysixfour wrote: post #15

I thought the purpose of these things was not to be fast, but to be able to run small models with very little power usage? I have a newer AMD laptop with an NPU, and my power usage doesn't change using the video effects that supposedly run on it, but goes up when using the nvidia studio effects. It seems like the NPUs are for very optimized models that do small tasks, like eye contact, background blur, autocorrect mo…

You're absolutely right IMO, given what I heard when launching on-device speech recognition on Pixel, and after leaving Google, what I see from ex. Apple Neural Engine vs. CPU when running ONNX stuff.

I'm a bit suspicious of the article's specific conclusion, because it is Qualcomm's ONNX, and it be out of date. Also, Android loved talking shit about Qualcomm software engineering.

That being said, its directionally correct, insomuch as consumer hardware AI acceleration claims are near-universally BS unless you're A) writing 1P software B) someone in the 1P really wants you to take advantage.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#34

Post by errantspark »

boomskats wrote: post #30
eightysixfour wrote: post #15

I thought the purpose of these things was not to be fast, but to be able to run small models with very little power usage? I have a newer AMD laptop with an NPU, and my power usage doesn't change using the video effects that supposedly run on it, but goes up when using the nvidia studio effects. It seems like the NPUs are for very optimized models that do small tasks, like eye contact, background blur, autocorrect mo…

That's especially true because yours is a Xilinx FPGA. The one that they just attached to the latest gen mobile ryzens is 5x more capable too. AMD are doing some fantastic work at the moment, they just don't seem to be shouting about it. This one is particularly interesting https://lore.kernel.org/lkml/DM6PR12MB3993D5ECA50B27682AEBE1... edit: not an FPGA. TIL. :'(

Wait sorry back up a bit here. I can buy a laptop that has a daughter FPGA in it? Does it have GPIO??? Are we seriously building hardware worth buying again in 2024? Do you have a link?

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#35

Post by beeflet »

boomskats wrote: post #30
eightysixfour wrote: post #15

I thought the purpose of these things was not to be fast, but to be able to run small models with very little power usage? I have a newer AMD laptop with an NPU, and my power usage doesn't change using the video effects that supposedly run on it, but goes up when using the nvidia studio effects. It seems like the NPUs are for very optimized models that do small tasks, like eye contact, background blur, autocorrect mo…

That's especially true because yours is a Xilinx FPGA. The one that they just attached to the latest gen mobile ryzens is 5x more capable too. AMD are doing some fantastic work at the moment, they just don't seem to be shouting about it. This one is particularly interesting https://lore.kernel.org/lkml/DM6PR12MB3993D5ECA50B27682AEBE1... edit: not an FPGA. TIL. :'(

It would be cool if most PCs had a general purpose FPGA that could be repurposed by the operating system. For example you could use it as a security processor like a TPM or as a bootrom, or you could repurpose it for DSP or something.

It just seems like this would be better in terms of firmware/security/bootloading because you would be more able to fix it if an exploit gets discovered, and it would be leaner because different operating systems can implement their own stuff (for example linux might not want pluton in-chip security, windows might not want coreboot or linux-based boot, bare metal applications can have much simpler boot).

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#36

Post by Kon-Peki »

jsheard wrote: post #8

These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of the Snapdragon X which isolates the NPU specifically but AMDs equivalent with the same ~50 TOPS performance target can be seen here, and takes up about as much area as three high performance CPU cores: https://www.techpowerup.com/325035/amd-strix-point-silicon…

Modern chips have to dedicate a certain percentage of the die to dark silicon [1] (or else they melt/throttle to uselessness), and these kinds of components count towards that amount. So the point of these components is to be used, but not to be used too much.

Instead of an NPU, they could have used those transistors and die space for any number of things. But they wouldn't have put additional high performance CPU cores there - that would increase the power density too much and cause thermal issues that can only be solved with permanent throttling.

[1] https://en.wikipedia.org/wiki/Dark_silicon

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#38

Post by moffkalast »

eightysixfour wrote: post #15

I thought the purpose of these things was not to be fast, but to be able to run small models with very little power usage? I have a newer AMD laptop with an NPU, and my power usage doesn't change using the video effects that supposedly run on it, but goes up when using the nvidia studio effects. It seems like the NPUs are for very optimized models that do small tasks, like eye contact, background blur, autocorrect mo…

[flagged]

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#39

Post by ezst »

jsheard wrote: post #8

These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of the Snapdragon X which isolates the NPU specifically but AMDs equivalent with the same ~50 TOPS performance target can be seen here, and takes up about as much area as three high performance CPU cores: https://www.techpowerup.com/325035/amd-strix-point-silicon…

I can't wait for the LLM fad to be over so we get some sanity (and efficiency) back. I personally have no use for this extra hardware ("GenAI" doesn't help me in any way nor supports any work-related tasks). Worse, most people have no use for that (and recent surveys even show predominant hostility towards AI creep). We shouldn't be paying extra for that, it should be opt-in, and then it would become clear (by looking at the sales and how few are willing to pay a premium for "AI") how overblown and unnecessary this is.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#40

Post by eightysixfour »

errantspark wrote: post #34
boomskats wrote: post #30

Earlier quoted context omitted.

That's especially true because yours is a Xilinx FPGA. The one that they just attached to the latest gen mobile ryzens is 5x more capable too. AMD are doing some fantastic work at the moment, they just don't seem to be shouting about it. This one is particularly interesting https://lore.kernel.org/lkml/DM6PR12MB3993D5ECA50B27682AEBE1... edit: not an FPGA. TIL. :'(

Wait sorry back up a bit here. I can buy a laptop that has a daughter FPGA in it? Does it have GPIO??? Are we seriously building hardware worth buying again in 2024? Do you have a link?

It isn't as fun as you think - they are setup for specific use cases and quite small. Here's a link to the software page: https://ryzenai.docs.amd.com/en/latest/index.html

The teeny-tiny "NPU," which is actually an FPGA, is 10 TOPS.

Edit: I've been corrected, not an FPGA, just an IP block from Xilinx.

Post reply on HN