Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI PCs Aren't Good at AI: The CPU Beats the NPU

github.com

Post reply on HN
Print view View original

51–60 of 303 posts

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#51

Post by renewiltord »

ezst wrote: post #39
jsheard wrote: post #8

These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of the Snapdragon X which isolates the NPU specifically but AMDs equivalent with the same ~50 TOPS performance target can be seen here, and takes up about as much area as three high performance CPU cores: https://www.techpowerup.com/325035/amd-strix-point-silicon…

I can't wait for the LLM fad to be over so we get some sanity (and efficiency) back. I personally have no use for this extra hardware ("GenAI" doesn't help me in any way nor supports any work-related tasks). Worse, most people have no use for that (and recent surveys even show predominant hostility towards AI creep). We shouldn't be paying extra for that, it should be opt-in, and then it would become clear (by lookin…

I was telling someone this and they gave me link to a laptop with higher battery life and better performance than my own, but I kept explaining to them that the feature I cared most about was die size. They couldn't understand it so I just had to leave them alone. Non-technical people don't get it. Die size is what I care about. It's a critical feature and so many mainstream companies are missing out on my money because they won't optimize die size. Disgusting.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#52

Post by wtallis »

moffkalast wrote: post #10
dmitrygr wrote: post #5

In general MAC unit utilization tends to be low for transformers, but 1.3% seems pretty bad. I wonder if they fucked up the memory interface for the NPU. All the MACs in the world are useless if you cannot feed them.

I recall looking over the Ryzen AI architecture and the NPU is just plugged into PCIe and thus gets completely crap memory bandwidth. I would expect it might be similar here.

It's unlikely to be literally connected over PCIe when it's on the same chip. It just looks like it's connected over PCIe because that's how you make peripherals discoverable to the OS. The integrated GPU also appears to be connected over PCIe, but obviously has access to far more memory bandwidth.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#53

Post by IshKebab »

Kon-Peki wrote: post #36
jsheard wrote: post #8

These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of the Snapdragon X which isolates the NPU specifically but AMDs equivalent with the same ~50 TOPS performance target can be seen here, and takes up about as much area as three high performance CPU cores: https://www.techpowerup.com/325035/amd-strix-point-silicon…

Modern chips have to dedicate a certain percentage of the die to dark silicon [1] (or else they melt/throttle to uselessness), and these kinds of components count towards that amount. So the point of these components is to be used, but not to be used too much. Instead of an NPU, they could have used those transistors and die space for any number of things. But they wouldn't have put additional high performance CPU co…

If they aren't being used it would be better to dedicate the space to more SRAM.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#55

Post by almostgotcaught »

boomskats wrote: post #49
wtallis wrote: post #44

Earlier quoted context omitted.

It's not a FPGA. It's an NPU IP block from the Xilinx side of the company. It was presumably originally developed to be run on a Xilinx FPGA, but that doesn't mean AMD did the stupid thing and actually fabbed a FPGA fabric instead of properly synthesizing the design for their laptop ASIC. Xilinx involvement does not automatically mean it's an FPGA.

Do you have any more reading on this? How come the XDNA drivers depend on Xilinx' XRT runtime?

because XRT has a plugin architecture: XRT<-shim plugin<-kernel driver. The shims register themselves with XRT. The XDNA driver repo houses both the shim and the kernel driver.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#56

Post by almostgotcaught »

pclmulqdq wrote: post #45
boomskats wrote: post #30

Earlier quoted context omitted.

That's especially true because yours is a Xilinx FPGA. The one that they just attached to the latest gen mobile ryzens is 5x more capable too. AMD are doing some fantastic work at the moment, they just don't seem to be shouting about it. This one is particularly interesting https://lore.kernel.org/lkml/DM6PR12MB3993D5ECA50B27682AEBE1... edit: not an FPGA. TIL. :'(

It's not an FPGA. It's a VLIW DSP that Xilinx built to go into an FPGA-SoC to help run ML models.

this is the correct answer. one of the compilers for this DSP is https://github.com/Xilinx/llvm-aie.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#57

Post by almostgotcaught »

boomskats wrote: post #47
eightysixfour wrote: post #40

Earlier quoted context omitted.

It isn't as fun as you think - they are setup for specific use cases and quite small. Here's a link to the software page: https://ryzenai.docs.amd.com/en/latest/index.html The teeny-tiny "NPU," which is actually an FPGA, is 10 TOPS. Edit: I've been corrected, not an FPGA, just an IP block from Xilinx.

Yes, the one on the ryzen 7000 chips like the 7840u isn't massive, but that's the last gen model. The one they've just released with the HX370 chip is estimated at 50 TOPS, which is better than Qualcomm's ARM flagship that this post is about. It's a fivefold improvement in a single generation, it's pretty exciting. A̵n̵d̵ ̵i̵t̵'̵s̵ ̵a̵n̵ ̵F̵P̵G̵A̵ It's not an FPGA

> And it's an FPGA.

nope it's not.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#58

Post by Janicc »

DrillShopper wrote: post #43
jamesy0ung wrote: post #14

What exactly does Windows do with a NPU? I don't own an 'AI PC' but it seems like the NPUs are slow and can't run much. I know Apple's Neural Engine is used to power Face ID and the facial recognition stuff in Photos, among other things.

It supports Microsoft's Recall (now required) spyware

Please remind me again how Recall sends data to Microsoft. I must've missed that part. Or are you against the print screen button too? I heard that takes images too. Very scary.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#59

Post by dekhn »

errantspark wrote: post #34
boomskats wrote: post #30

Earlier quoted context omitted.

That's especially true because yours is a Xilinx FPGA. The one that they just attached to the latest gen mobile ryzens is 5x more capable too. AMD are doing some fantastic work at the moment, they just don't seem to be shouting about it. This one is particularly interesting https://lore.kernel.org/lkml/DM6PR12MB3993D5ECA50B27682AEBE1... edit: not an FPGA. TIL. :'(

Wait sorry back up a bit here. I can buy a laptop that has a daughter FPGA in it? Does it have GPIO??? Are we seriously building hardware worth buying again in 2024? Do you have a link?

If you want GPIOs, you don't need (or want) an FPGA.

I don't know the details of your use case, but I work with low level hardware driven by GPIOs and after a bit of investigation, concluded that having direect GPIO access in a modern PC was not necessary or desirable compared to the alternatives.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#60

Post by wtallis »

boomskats wrote: post #49
wtallis wrote: post #44

Earlier quoted context omitted.

It's not a FPGA. It's an NPU IP block from the Xilinx side of the company. It was presumably originally developed to be run on a Xilinx FPGA, but that doesn't mean AMD did the stupid thing and actually fabbed a FPGA fabric instead of properly synthesizing the design for their laptop ASIC. Xilinx involvement does not automatically mean it's an FPGA.

Do you have any more reading on this? How come the XDNA drivers depend on Xilinx' XRT runtime?

It would be surprising and strange if AMD didn't reuse the software framework they've already built for doing AI when that IP block is instantiated on an FPGA fabric rather than hardened in an ASIC.
Post reply on HN