Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI PCs Aren't Good at AI: The CPU Beats the NPU

github.com

Post reply on HN
Print view View original

171–180 of 303 posts

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#171

Post by fijiaarone »

renewiltord wrote: post #51
ezst wrote: post #39

Earlier quoted context omitted.

I can't wait for the LLM fad to be over so we get some sanity (and efficiency) back. I personally have no use for this extra hardware ("GenAI" doesn't help me in any way nor supports any work-related tasks). Worse, most people have no use for that (and recent surveys even show predominant hostility towards AI creep). We shouldn't be paying extra for that, it should be opt-in, and then it would become clear (by lookin…

I was telling someone this and they gave me link to a laptop with higher battery life and better performance than my own, but I kept explaining to them that the feature I cared most about was die size. They couldn't understand it so I just had to leave them alone. Non-technical people don't get it. Die size is what I care about. It's a critical feature and so many mainstream companies are missing out on my money beca…

Yeah, I know what you mean. I hate lugging around a big CPU core.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#172

Post by MBCook »

Terr_ wrote: post #166
MBCook wrote: post #153

Earlier quoted context omitted.

Is there demand? Or do they just assume there is? If they shove it in every single product and that’s all anyone advertises, whether consumers know it will help them or not, you don’t get a lot of choice. If you want the latest chip, you’re getting AI stuff. That’s all there is to it.

"The math is clear: 100% of our our car sales come from models with our company logo somewhere on the front, which shows incredible customer desire for logos. We should consider offering a new luxury trim level with more of them." "How many models to we have without logos?" "Huh? Why would we do that?"

Heh. Yeah more or less.

To some degree I understand it, because as we’ve all noticed computers have pretty much plateaued for the average person. They last much longer. You don’t need to replace them every two years anymore because the software isn’t out stripping them so fast.

AI is the first thing to come along in quite a while that not only needs significant power but it’s just something different. It’s something they can say your old computer doesn’t have that the new one does. Other than being 5% faster or whatever.

So even if people don’t need it, and even if they notice they don’t need it, it’s something to market on.

The stuff up thread about it being the hotness that Wall Street loves is absolutely a thing too.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#173

Post by bugbuddy »

nxobject wrote: post #165
cj wrote: post #163

Earlier quoted context omitted.

Rightly so. The M processor really did completely eliminate all sense of “lag” for basic computing (web browsing, restarting your computer, etc). Everything happens nearly instantly, even on the first generation M1 processor. The experience of “waiting for something to load” went away. Not to mention these machines easily last 5-10 years.

As a very happy M1 Max user (should've shelled out for 64GB of RAM, though, for local LLMs!), I don't look forward to seeing how the Google Workspace/Notions/etc. of the world somehow reintroduce lag back in.

The problem for Intel and AMD is they are stuck with an OS that ships with a lag-inducing Anti-malware suite. I just did a simple git log and it took 2000% longer than usual because the Antivirus was triggered to scan and run a simulation on each machine instruction and byte of data accessed. The commit log window stayed blank waiting to load long enough for me to complete another tiny project. It always ruin my day.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#174

Post by kalleboo »

moffkalast wrote: post #50
eightysixfour wrote: post #42

Earlier quoted context omitted.

The 7940HS shipped before recall and doesn't support it because it is not performant enough, so, that doesn't make sense. I just gave you a use case, mine in particular uses it for background blur and eye contact filters with the webcam and uses essentially no power to do it. If I do the same filters with nvidia broadcast, the power usage is dramatically higher.

I doubt there's no notable power draw, NPUs in general have always pulled a handful of watts which should at least about match a modern CPU's idle draw. But it does seem odd that your power usage doesn't change at all, it might be always powered on or something. Eye contact filters seem like a horrible thing, autocorrect won't work better than a dictionary with a tiny model and I doubt these things can come even clos…

> I doubt these things can come even close to running whisper for decent voice transcription

https://github.com/ggerganov/whisper.cpp/pull/566

"The performance gain is more than x3 compared to 8-thread CPU"

And this is on the 3 year old M1 Pro

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#175

Post by errantspark »

dekhn wrote: post #59
errantspark wrote: post #34

Earlier quoted context omitted.

Wait sorry back up a bit here. I can buy a laptop that has a daughter FPGA in it? Does it have GPIO??? Are we seriously building hardware worth buying again in 2024? Do you have a link?

If you want GPIOs, you don't need (or want) an FPGA. I don't know the details of your use case, but I work with low level hardware driven by GPIOs and after a bit of investigation, concluded that having direect GPIO access in a modern PC was not necessary or desirable compared to the alternatives.

I get a lot of use out of the PRUs on the BeagleboneBlack, I would absolutely get use out of an FPGA in a laptop.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#176

Post by justahuman74 »

Spooky23 wrote: post #162
brookst wrote: post #145

Earlier quoted context omitted.

The shoehorning only works if there is buyer demand. As a company, if customers are willing to pay a premium for a NPU, or if they are unwilling to buy a product without one, it is not your place to say “hey we don’t really believe in the AI hype so we’re going to sell products people don’t want to prove a point”

Apple will have a completely AI capable product line in 18 months, with the major platforms basically done. Microsoft is built around the broken Intel tick/tick model of incremental improvement — they are stuck with OEM shitware that will take years to flush out of the channel. That means for AI, they are stuck with cloud based OpenAI, where NVIDIA has them by the balls and the hyperscalers are all fighting for GPU.…

Who is getting $400/y of value from that?

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#177

Post by nxobject »

Spooky23 wrote: post #162
brookst wrote: post #145

Earlier quoted context omitted.

The shoehorning only works if there is buyer demand. As a company, if customers are willing to pay a premium for a NPU, or if they are unwilling to buy a product without one, it is not your place to say “hey we don’t really believe in the AI hype so we’re going to sell products people don’t want to prove a point”

Apple will have a completely AI capable product line in 18 months, with the major platforms basically done. Microsoft is built around the broken Intel tick/tick model of incremental improvement — they are stuck with OEM shitware that will take years to flush out of the channel. That means for AI, they are stuck with cloud based OpenAI, where NVIDIA has them by the balls and the hyperscalers are all fighting for GPU.…

I hope that once they get a baseline level of AI functionality in, they start working with larger LLMs to enable some form of RAG... that might be their next generational shift.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#178

Post by JohnFen »

heavyset_go wrote: post #120
JohnFen wrote: post #67

Earlier quoted context omitted.

> These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. This has been my thinking. Today you have to go out of your way to buy a system with an NPU, so I don't have any. But tomorrow, will they just be included by default? That seems like a waste for those of us who aren't going to be running models. I wonder what other uses they could be put…

The idea is that your OS and apps will integrate ML models, so you will be running models whether you know it or not.

I'm confident that I'll be able to know and control whether or not my Linux and BSD machines will be using ML models.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#179

Post by pclmulqdq »

AlexandrB wrote: post #16
isusmelj wrote: post #13

I think the results show that just in general the compute is not used well. That the CPU took 8.4ms and GPU took 3.2ms shows a very small gap. I'd expect more like 10x - 20x difference here. I'd assume that the onnxruntime might be the issue. I think some hardware vendors just release the compute units without shipping proper support yet. Let's see how fast that will change. Also, people often mistake the reason for…

> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. I have a sneaking suspicion that the real real reason for an NPU is marketing. "Oh look, NVDA is worth $3.3T - let's make sure we stick some AI stuff in our products too."

The correct way to make a true "NPU" is to 10x your memory bandwidth and feed a regular old multicore CPU with SIMD/vector instructions (and maybe a matrix multiply unit).

Most of these small NPUs are actually made for CNNs and other models where "stream data through weights" applies. They have a huge speedup there. When you stream weights across data (any LLM or other large model), you are almost certain to be bound by memory bandwidth.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#180

Post by Grimblewald »

_davide_ wrote: post #167

The RTX 4080 should be capable of ~40 TFLOPS, yet they only report 2,160 billion operations per second. Shouldn't this be enough to reconsider the benchmark? They probably made some serious error in measuring FLOPS. Regarding the fact that CPU beats NPU is possible but they should benchmark many matrix multiplications without any application synchronization in order to have a decent comparison.

That isnt the half of it. A quick skim of the documentation shows that the cpu inference wasnt done in a comparable way either.
Post reply on HN