Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI PCs Aren't Good at AI: The CPU Beats the NPU

github.com

Post reply on HN
Print view View original

281–290 of 303 posts

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#281

Post by johnny22 »

noduerme wrote: post #233
rerdavies wrote: post #223

Earlier quoted context omitted.

Why would the two not be entirely comparable? PyTorch may be slower at building the models; but once the model is compiled and loaded on the NPU, there's just not a whole lot of Python involved anymore. A few hundred CPU cycles to push the input data using python; a few hundred CPU cycles to receive the results using python. And everything in-between gets executed on the NPU.

I really wish Python wasn't the language controlling all the C code. You need a controller, in a scripting language that's easy to modify, but it's a rather hideous choice. It would be like choosing to build the world's largest social network in PHP or something. lol.

isn't that the case? Which then became a dialect of php with a custom interpreter (and then compiler) as they scaled.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#282

Post by fvv »

caeril wrote: post #264
theGnuMe wrote: post #261

Earlier quoted context omitted.

There are clean room approaches like AMDs and Scale.

Geohot has multiple (and ongoing) rants about the sheer instability of AMD RDNA3 drivers. Lisa Su engaged directly with him on this, and she didn't seem to give a shit about their problems. AMD is not taking ML applications seriously, outside of their marketing hype.

Rdna3 is not cdna

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#283

Post by ddingus »

lynguist wrote: post #193
ddingus wrote: post #181

Earlier quoted context omitted.

I have a first gen M1 and it holds up very nicely even today. I/O is crazy fast and high compute loads get done efficiently. One can bury the machine and lose very little basic interactivity. That part users really like. Frankly the only downside of the MacBook Air is the tiny storage. The 8GB RAM is actually enough most of the time. But general system storage with only 1/4 TB is cramped consistently. Been thinking a…

Why did you buy a 256GB device for personal use in the first place? Too good of a deal? Or saving these $400 for upgrades for something else?

I got it for a song. Literally a coupla hundred bucks a few months after release.

So yeah, great deal. And I really wanted to run the new CPU.

Frankly, I can do more and generally faster than I would expect running on those limited resources. It has been a quite nice surprise.

For a lot of what I do, the RAM and storage are enough.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#284

Post by ddingus »

112233 wrote: post #227
lynguist wrote: post #193

Earlier quoted context omitted.

Why did you buy a 256GB device for personal use in the first place? Too good of a deal? Or saving these $400 for upgrades for something else?

Not OP, but by booting M1 from external thunderbolt nvme you lose less than 50% of benchmark disk throughput (3GB/s is still ridiculously fast), can buy 8TB drive for less than 1k, plus can boot it on another M1 mac if something happens. If there was "max mem, min disk" model, would def get that.

Interesting. You know I bought one of those USB 3 port expanders from TEMU and it is excellent! (I know, TEMU right? But it was so cheap!)

I could 3d print a couple of brackets and probably lodge a bigger SSD or the smaller form factor eMMC I think and pack it all into a little package one just plugs in. The port extender is currently shaped such that it fits right under the Air tilting it nicely for general use.

The Air only has external USB... still, I don't need to boot from it. The internal one can continue to do that. Storage is storage for most tasks.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#285

Post by barfingclouds »

ezst wrote: post #39
jsheard wrote: post #8

These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of the Snapdragon X which isolates the NPU specifically but AMDs equivalent with the same ~50 TOPS performance target can be seen here, and takes up about as much area as three high performance CPU cores: https://www.techpowerup.com/325035/amd-strix-point-silicon…

I can't wait for the LLM fad to be over so we get some sanity (and efficiency) back. I personally have no use for this extra hardware ("GenAI" doesn't help me in any way nor supports any work-related tasks). Worse, most people have no use for that (and recent surveys even show predominant hostility towards AI creep). We shouldn't be paying extra for that, it should be opt-in, and then it would become clear (by lookin…

Agreed. I use perplexity and chat gpt daily, but if I understand correctly that’s all done off device which is fine with me. I don’t need to generate weird images, or have my phone rewrite emails for me, or summarize stuff.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#286

Post by heavyset_go »

JohnFen wrote: post #178
heavyset_go wrote: post #120

Earlier quoted context omitted.

The idea is that your OS and apps will integrate ML models, so you will be running models whether you know it or not.

I'm confident that I'll be able to know and control whether or not my Linux and BSD machines will be using ML models.

I agree with the premise as a Linux user myself, but if you're using any JetBrains products, or Zoom, you're running models on the client-side. I suspect small models will continue to creep into apps. Even Firefox ships ML models in the browser.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#287

Post by heavyset_go »

hollerith wrote: post #186
JohnFen wrote: post #178

Earlier quoted context omitted.

I'm confident that I'll be able to know and control whether or not my Linux and BSD machines will be using ML models.

--and whether anyone is using your interactions with your computer to train a model.

Luckily, while NPUs do nothing about data exfiltration, they're a poor solution for training models. Your data is still going to get sucked up to the mothership, but offloading training to your machine hopefully won't happen.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#288

Post by wkat4242 »

Spooky23 wrote: post #159
AlexandrB wrote: post #16

Earlier quoted context omitted.

> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. I have a sneaking suspicion that the real real reason for an NPU is marketing. "Oh look, NVDA is worth $3.3T - let's make sure we stick some AI stuff in our products too."

Microsoft needs to throw something in the gap to slow down MacBook attrition. The M processors changed the game. My teams support 250k users. I went from 50 MacBooks in 2020 to over 10,000 today. I added zero staff - we manage them like iPhones.

In our company we see the opposite. 5 years ago all the devs wanted Mac instead of Linux. Now they want to go back.

I think part of the reason is that we manage Mac pretty strictly now but we're getting there with Linux too.

We also tried to get them to use WSL 1 and 2 but they just laugh at it :) And point at its terrible disk performance and other dealbreakers. Can't blame them.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#289

Post by wkat4242 »

Spooky23 wrote: post #162
brookst wrote: post #145

Earlier quoted context omitted.

The shoehorning only works if there is buyer demand. As a company, if customers are willing to pay a premium for a NPU, or if they are unwilling to buy a product without one, it is not your place to say “hey we don’t really believe in the AI hype so we’re going to sell products people don’t want to prove a point”

Apple will have a completely AI capable product line in 18 months, with the major platforms basically done. Microsoft is built around the broken Intel tick/tick model of incremental improvement — they are stuck with OEM shitware that will take years to flush out of the channel. That means for AI, they are stuck with cloud based OpenAI, where NVIDIA has them by the balls and the hyperscalers are all fighting for GPU.…

> while Office 365 AI is like $400+ a year per user

And I'm pretty sure this is only Introductory pricing. As people get used to it and use it more it won't cover the cost. I think they rely on the gym model currently; many people not using the ai features much. But eventually that will change. Also, many companies figured that out and pull the copilot license from users that don't use it enough.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#290

Post by noduerme »

johnny22 wrote: post #281
noduerme wrote: post #233

Earlier quoted context omitted.

I really wish Python wasn't the language controlling all the C code. You need a controller, in a scripting language that's easy to modify, but it's a rather hideous choice. It would be like choosing to build the world's largest social network in PHP or something. lol.

isn't that the case? Which then became a dialect of php with a custom interpreter (and then compiler) as they scaled.

Yes, that was the case. I was being sarcastic. Zuck wrote facebook in PHP and spent millions of dollars then writing a custom interpreter to let his janky code run slightly faster than normal.
Post reply on HN