Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI PCs Aren't Good at AI: The CPU Beats the NPU

github.com

Post reply on HN
Print view View original

161–170 of 303 posts

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#161

Post by jcgrillo »

AlotOfReading wrote: post #154
jcgrillo wrote: post #147

Earlier quoted context omitted.

Question--what's to be lost by making your features sufficiently not dense to allow them to cool at full tilt?

Messes with timing, among other things. A lot of those structures are relatively fixed blocks that are designed for specific sizes. Signals take more time to propagate longer distances, and longer conductors have worse properties. Dense and hot is faster and more broadly useful.

Interesting, so does that mean we're basically out of runway without aggressive cooling?

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#162

Post by Spooky23 »

brookst wrote: post #145
itishappy wrote: post #23

Earlier quoted context omitted.

I assume you're both right. I'm sure NPUs exist to fill a very real niche, but I'm also sure they're being shoehorned in everywhere regardless of product fit because "AI big right now."

The shoehorning only works if there is buyer demand. As a company, if customers are willing to pay a premium for a NPU, or if they are unwilling to buy a product without one, it is not your place to say “hey we don’t really believe in the AI hype so we’re going to sell products people don’t want to prove a point”

Apple will have a completely AI capable product line in 18 months, with the major platforms basically done.

Microsoft is built around the broken Intel tick/tick model of incremental improvement — they are stuck with OEM shitware that will take years to flush out of the channel. That means for AI, they are stuck with cloud based OpenAI, where NVIDIA has them by the balls and the hyperscalers are all fighting for GPU.

Apple will deliver local AI features as software (the hardware is “free”) at a much higher margin - while Office 365 AI is like $400+ a year per user.

You’ll have people getting iPhones to get AI assisted emails or whatever Apple does that is useful.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#163

Post by cj »

Spooky23 wrote: post #159
AlexandrB wrote: post #16

Earlier quoted context omitted.

> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. I have a sneaking suspicion that the real real reason for an NPU is marketing. "Oh look, NVDA is worth $3.3T - let's make sure we stick some AI stuff in our products too."

Microsoft needs to throw something in the gap to slow down MacBook attrition. The M processors changed the game. My teams support 250k users. I went from 50 MacBooks in 2020 to over 10,000 today. I added zero staff - we manage them like iPhones.

Rightly so.

The M processor really did completely eliminate all sense of “lag” for basic computing (web browsing, restarting your computer, etc). Everything happens nearly instantly, even on the first generation M1 processor. The experience of “waiting for something to load” went away.

Not to mention these machines easily last 5-10 years.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#164

Post by conradev »

AlexandrB wrote: post #16
isusmelj wrote: post #13

I think the results show that just in general the compute is not used well. That the CPU took 8.4ms and GPU took 3.2ms shows a very small gap. I'd expect more like 10x - 20x difference here. I'd assume that the onnxruntime might be the issue. I think some hardware vendors just release the compute units without shipping proper support yet. Let's see how fast that will change. Also, people often mistake the reason for…

> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. I have a sneaking suspicion that the real real reason for an NPU is marketing. "Oh look, NVDA is worth $3.3T - let's make sure we stick some AI stuff in our products too."

The real consumers of the NPUs are the operating systems themselves. Google’s TPU and Apple’s ANE are used to power OS features like Apple’s Face ID and Google’s image enhancements.

We’re seeing these things in traditional PCs now because Microsoft has demanded it so that Microsoft can use it in Windows 11.

Any use by third party software is a lower priority

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#165

Post by nxobject »

cj wrote: post #163
Spooky23 wrote: post #159

Earlier quoted context omitted.

Microsoft needs to throw something in the gap to slow down MacBook attrition. The M processors changed the game. My teams support 250k users. I went from 50 MacBooks in 2020 to over 10,000 today. I added zero staff - we manage them like iPhones.

Rightly so. The M processor really did completely eliminate all sense of “lag” for basic computing (web browsing, restarting your computer, etc). Everything happens nearly instantly, even on the first generation M1 processor. The experience of “waiting for something to load” went away. Not to mention these machines easily last 5-10 years.

As a very happy M1 Max user (should've shelled out for 64GB of RAM, though, for local LLMs!), I don't look forward to seeing how the Google Workspace/Notions/etc. of the world somehow reintroduce lag back in.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#166

Post by Terr_ »

MBCook wrote: post #153
brookst wrote: post #145

Earlier quoted context omitted.

The shoehorning only works if there is buyer demand. As a company, if customers are willing to pay a premium for a NPU, or if they are unwilling to buy a product without one, it is not your place to say “hey we don’t really believe in the AI hype so we’re going to sell products people don’t want to prove a point”

Is there demand? Or do they just assume there is? If they shove it in every single product and that’s all anyone advertises, whether consumers know it will help them or not, you don’t get a lot of choice. If you want the latest chip, you’re getting AI stuff. That’s all there is to it.

"The math is clear: 100% of our our car sales come from models with our company logo somewhere on the front, which shows incredible customer desire for logos. We should consider offering a new luxury trim level with more of them."

"How many models to we have without logos?"

"Huh? Why would we do that?"

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#167

Post by _davide_ »

The RTX 4080 should be capable of ~40 TFLOPS, yet they only report 2,160 billion operations per second. Shouldn't this be enough to reconsider the benchmark? They probably made some serious error in measuring FLOPS. Regarding the fact that CPU beats NPU is possible but they should benchmark many matrix multiplications without any application synchronization in order to have a decent comparison.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#168

Post by james_marks »

jcgrillo wrote: post #138
ezst wrote: post #39

Earlier quoted context omitted.

I can't wait for the LLM fad to be over so we get some sanity (and efficiency) back. I personally have no use for this extra hardware ("GenAI" doesn't help me in any way nor supports any work-related tasks). Worse, most people have no use for that (and recent surveys even show predominant hostility towards AI creep). We shouldn't be paying extra for that, it should be opt-in, and then it would become clear (by lookin…

I just got an iphone and the whole photos thing is absolutely garbage. All I wanted to do was look through my damn photos and find one I took recently but it started playing some random music and organized them in no discernible order.. like it wasn't the reverse time sorted.. Idk what kind of fucked up "creative process" came up with that bullshit but I sure wish they'd unfuck it stat. The camera is real good though…

There’s an album called “Recents” that’s chronological and scrolled to the end.

“Recent” seems to mean everything; I’ve got 6k+ photos, I think since the last fresh install, which is many devices ago.

Sounds like the view you’re looking for and will stick as the default once you find it, but you do have to bat away some BS at first.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#169

Post by singlepaynews »

thfuran wrote: post #73
nl wrote: post #62

Earlier quoted context omitted.

Is this a parody? Why would anyone care about die size? And if you do why not get one of the many low power laptops with Atoms etc that do have small die size?

Yes, they're making fun of the comment they replied to.

Would you do me the favor of explaining the joke? I get the premise—nobody cares about die size, but the comment being mocked seems perfectly innocuous to me? They want a laptop without an NPU b/c according to link we get more out of CPU anyways? What am I missing here?
Post reply on HN