AI PCs Aren't Good at AI: The CPU Beats the NPU

Post by kristianp » Wed, Oct 16, 2024, 10:37 PM UTC

refulgentis wrote: post #32

eightysixfour wrote: post #15

I thought the purpose of these things was not to be fast, but to be able to run small models with very little power usage? I have a newer AMD laptop with an NPU, and my power usage doesn't change using the video effects that supposedly run on it, but goes up when using the nvidia studio effects. It seems like the NPUs are for very optimized models that do small tasks, like eye contact, background blur, autocorrect mo…

You're absolutely right IMO, given what I heard when launching on-device speech recognition on Pixel, and after leaving Google, what I see from ex. Apple Neural Engine vs. CPU when running ONNX stuff. I'm a bit suspicious of the article's specific conclusion, because it is Qualcomm's ONNX, and it be out of date. Also, Android loved talking shit about Qualcomm software engineering. That being said, its directionally c…

Post by boomskats » Wed, Oct 16, 2024, 10:43 PM UTC

almostgotcaught wrote: post #55

boomskats wrote: post #49

Earlier quoted context omitted.

Do you have any more reading on this? How come the XDNA drivers depend on Xilinx' XRT runtime?

because XRT has a plugin architecture: XRT<-shim plugin<-kernel driver. The shims register themselves with XRT. The XDNA driver repo houses both the shim and the kernel driver.

Post by walterbell » Wed, Oct 16, 2024, 10:44 PM UTC

beeflet wrote: post #35

boomskats wrote: post #30

Earlier quoted context omitted.

That's especially true because yours is a Xilinx FPGA. The one that they just attached to the latest gen mobile ryzens is 5x more capable too. AMD are doing some fantastic work at the moment, they just don't seem to be shouting about it. This one is particularly interesting https://lore.kernel.org/lkml/DM6PR12MB3993D5ECA50B27682AEBE1... edit: not an FPGA. TIL. :'(

It would be cool if most PCs had a general purpose FPGA that could be repurposed by the operating system. For example you could use it as a security processor like a TPM or as a bootrom, or you could repurpose it for DSP or something. It just seems like this would be better in terms of firmware/security/bootloading because you would be more able to fix it if an exploit gets discovered, and it would be leaner because…

Post by boomskats » Wed, Oct 16, 2024, 10:46 PM UTC

wtallis wrote: post #60

boomskats wrote: post #49

Earlier quoted context omitted.

Do you have any more reading on this? How come the XDNA drivers depend on Xilinx' XRT runtime?

It would be surprising and strange if AMD didn't reuse the software framework they've already built for doing AI when that IP block is instantiated on an FPGA fabric rather than hardened in an ASIC.

Post by AlexAndScripts » Wed, Oct 16, 2024, 10:47 PM UTC

jsheard wrote: post #71

JohnFen wrote: post #67

Earlier quoted context omitted.

> These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. This has been my thinking. Today you have to go out of your way to buy a system with an NPU, so I don't have any. But tomorrow, will they just be included by default? That seems like a waste for those of us who aren't going to be running models. I wonder what other uses they could be put…

> But tomorrow, will they just be included by default? That's already the way things are going due to Microsoft decreeing that Copilot+ is the future of Windows, so AMD and Intel are both putting NPUs which meet the Copilot+ performance standard into every consumer part they make going forwards to secure OEM sales.