Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI PCs Aren't Good at AI: The CPU Beats the NPU

github.com

Post reply on HN
Print view View original

181–190 of 303 posts

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#181

Post by ddingus »

cj wrote: post #163
Spooky23 wrote: post #159

Earlier quoted context omitted.

Microsoft needs to throw something in the gap to slow down MacBook attrition. The M processors changed the game. My teams support 250k users. I went from 50 MacBooks in 2020 to over 10,000 today. I added zero staff - we manage them like iPhones.

Rightly so. The M processor really did completely eliminate all sense of “lag” for basic computing (web browsing, restarting your computer, etc). Everything happens nearly instantly, even on the first generation M1 processor. The experience of “waiting for something to load” went away. Not to mention these machines easily last 5-10 years.

I have a first gen M1 and it holds up very nicely even today. I/O is crazy fast and high compute loads get done efficiently.

One can bury the machine and lose very little basic interactivity. That part users really like.

Frankly the only downside of the MacBook Air is the tiny storage. The 8GB RAM is actually enough most of the time. But general system storage with only 1/4 TB is cramped consistently.

Been thinking about sending the machine out to one of those upgrade shops...

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#182

Post by JohnFen »

crazygringo wrote: post #99
JohnFen wrote: post #67

Earlier quoted context omitted.

> These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. This has been my thinking. Today you have to go out of your way to buy a system with an NPU, so I don't have any. But tomorrow, will they just be included by default? That seems like a waste for those of us who aren't going to be running models. I wonder what other uses they could be put…

Aren't they used for speech recognition -- for dictation? Also for FaceID. They're useful for more things than just LLM's.

Yes, but I'm not interested in those sorts of uses. I'm wondering what else an NPU could be used for. I don't know what an NPU actually is at a technical level, so I'm ignorant of the possibilities.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#183

Post by ddingus »

MBCook wrote: post #172
Terr_ wrote: post #166

Earlier quoted context omitted.

"The math is clear: 100% of our our car sales come from models with our company logo somewhere on the front, which shows incredible customer desire for logos. We should consider offering a new luxury trim level with more of them." "How many models to we have without logos?" "Huh? Why would we do that?"

Heh. Yeah more or less. To some degree I understand it, because as we’ve all noticed computers have pretty much plateaued for the average person. They last much longer. You don’t need to replace them every two years anymore because the software isn’t out stripping them so fast. AI is the first thing to come along in quite a while that not only needs significant power but it’s just something different. It’s something…

That was all true nearly 10 years ago. And it has only improved. Almost any computer one finds these days is capable of the basics.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#184

Post by hakfoo »

Spooky23 wrote: post #162
brookst wrote: post #145

Earlier quoted context omitted.

The shoehorning only works if there is buyer demand. As a company, if customers are willing to pay a premium for a NPU, or if they are unwilling to buy a product without one, it is not your place to say “hey we don’t really believe in the AI hype so we’re going to sell products people don’t want to prove a point”

Apple will have a completely AI capable product line in 18 months, with the major platforms basically done. Microsoft is built around the broken Intel tick/tick model of incremental improvement — they are stuck with OEM shitware that will take years to flush out of the channel. That means for AI, they are stuck with cloud based OpenAI, where NVIDIA has them by the balls and the hyperscalers are all fighting for GPU.…

We're still looking for "that is useful".

The stuff they've been trying to sell AI to the public with is increasingly looking as absurd as every 1978 "you'll store your recipes on the home computer" argument.

AI text became a Human Centipede story: Start with a coherent 10-word sentence, let AI balloon it into five pages of flowery nonsense, send it to someone else, who has their AI smash it back down to 10 meaningful words.

Coding assistance, even as spicy autocorrect, is often a net negative as you have to plow through hallucinations and weird guesses as to what you want but lack the tools to explain to it.

Image generation is already heading rapidly into cringe territory, in part due to some very public social media operations. I can imagine your kids' kids in 2040 finding out they generated AI images in the 2020s and looking at them with the same embarrassment you'd see if they dug out your high-school emo fursona.

There might well be some more "closed-loop" AI applications that make sense. But are they going to be running on every desktop in the world? Or are they going to be mostly used in datacentres and purpose-built embedded devices?

I also wonder how well some of the models and techniques scale down. I know Microsoft pushed a minimum spec to promote a machine as Copilot-ready, but that seems like it's going to be "Vista Basic Ready" redux as people try to run tools designed for datacentres full of Quadro cards, or at least high-end GPUs, on their $299 HP laptop.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#185

Post by kalleboo »

wmf wrote: post #149
kalleboo wrote: post #133

Earlier quoted context omitted.

> most people have no use for that Apple originally added their NPUs before the current LLM wave to support things like indexing your photo library so that objects and people are searchable. These features are still very popular. I don't think these NPUs are fast enough for GenAI anyway.

MS Copilot and "Apple Intelligence" are running a small language model and image generation on the NPU so that should count as "GenAI".

It's still in beta so we'll see how things go but I saw someone testing what Apple Intelligence ran on-device vs sent off to the "private secure cloud" and even stuff like text summaries were being sent to the cloud.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#186

Post by hollerith »

JohnFen wrote: post #178
heavyset_go wrote: post #120

Earlier quoted context omitted.

The idea is that your OS and apps will integrate ML models, so you will be running models whether you know it or not.

I'm confident that I'll be able to know and control whether or not my Linux and BSD machines will be using ML models.

--and whether anyone is using your interactions with your computer to train a model.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#187

Post by dekhn »

errantspark wrote: post #175
dekhn wrote: post #59

Earlier quoted context omitted.

If you want GPIOs, you don't need (or want) an FPGA. I don't know the details of your use case, but I work with low level hardware driven by GPIOs and after a bit of investigation, concluded that having direect GPIO access in a modern PC was not necessary or desirable compared to the alternatives.

I get a lot of use out of the PRUs on the BeagleboneBlack, I would absolutely get use out of an FPGA in a laptop.

It makes more sense to me to just use the BeagleboneBlack in concert with the FPGA. Unless you have highly specific compute or data movement needs that can't be satisfied over a USB serial link. If you have those needs, and you need a laptop, I guess an FPGA makes sense but that's a teeny market.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#188

Post by jjmarr »

hakfoo wrote: post #184
Spooky23 wrote: post #162

Earlier quoted context omitted.

Apple will have a completely AI capable product line in 18 months, with the major platforms basically done. Microsoft is built around the broken Intel tick/tick model of incremental improvement — they are stuck with OEM shitware that will take years to flush out of the channel. That means for AI, they are stuck with cloud based OpenAI, where NVIDIA has them by the balls and the hyperscalers are all fighting for GPU.…

We're still looking for "that is useful". The stuff they've been trying to sell AI to the public with is increasingly looking as absurd as every 1978 "you'll store your recipes on the home computer" argument. AI text became a Human Centipede story: Start with a coherent 10-word sentence, let AI balloon it into five pages of flowery nonsense, send it to someone else, who has their AI smash it back down to 10 meaningfu…

Cringe emo girls are trendy now because the nostalgia cycle is hitting the early 2000s. Your kid would be impressed if you told them you were a goth gf. It's not hard to imagine the same will happen with primitive AIs in the 40s.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#189

Post by zdw »

bugbuddy wrote: post #173
nxobject wrote: post #165

Earlier quoted context omitted.

As a very happy M1 Max user (should've shelled out for 64GB of RAM, though, for local LLMs!), I don't look forward to seeing how the Google Workspace/Notions/etc. of the world somehow reintroduce lag back in.

The problem for Intel and AMD is they are stuck with an OS that ships with a lag-inducing Anti-malware suite. I just did a simple git log and it took 2000% longer than usual because the Antivirus was triggered to scan and run a simulation on each machine instruction and byte of data accessed. The commit log window stayed blank waiting to load long enough for me to complete another tiny project. It always ruin my day.

This is most likely due to corporate malware.

Even modern macs can be brought to their knees by something that rhymes with FrowdStrike Calcon and interrupts all IO.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#190

Post by lincpa »

isusmelj wrote: post #13

I think the results show that just in general the compute is not used well. That the CPU took 8.4ms and GPU took 3.2ms shows a very small gap. I'd expect more like 10x - 20x difference here. I'd assume that the onnxruntime might be the issue. I think some hardware vendors just release the compute units without shipping proper support yet. Let's see how fast that will change. Also, people often mistake the reason for…

[dead]
Post reply on HN