Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI PCs Aren't Good at AI: The CPU Beats the NPU

github.com

Post reply on HN
Print view View original

261–270 of 303 posts

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#261

Post by theGnuMe »

david-gpu wrote: post #238
noduerme wrote: post #232

Earlier quoted context omitted.

I'm not sure why this is a moat. Isn't it just a matter of translation from CUDA to some other instruction set? If AMD or someone else makes cheaper hardware that does the same thing, it doesn't seem like a stretch for them to release a PyTorch patch or whatever.

Most of the computations are done inside NVidia proprietary libraries, not open-source CUDA. And if you saw what goes inside those libraries, I think you would agree that it is a substantial moat.

There are clean room approaches like AMDs and Scale.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#262

Post by consteval »

Hizonner wrote: post #239
cjbgkagh wrote: post #136

> We've tried to avoid that by making both the input matrices more square, so that tiling and reuse should be possible. While it might be possible it would not surprise me if a number of possible optimizations had not made it into Onnx. It appears that Qualcomm does not give direct access to the NPU and users are expected to use frameworks to convert models over to it, and in my experience conversion tools generally…

> While LLMs are the new hotness there are huge number of specialized tasks that small models are really useful for. Can you give some examples? Preferably examples that will run continuously enough for even a small model to stay in cache, and are valuable enough to a significant number of users to justify that cache footprint? I am not saying there aren't any, but I also honestly don't know what they are and would l…

iPhones use a lot of these. There's a bunch of little features that run on the NPU.

Suggestions, predictive text, smart image search, automatic image classification, text selection in images, image processing. These don't run continuously, but I think they are valuable to a lot of users. The predictive text is quite good, and it's very nice to be able to search for vague terms like "license plate" and get images in my camera roll. Plus, selecting text and copying it from images is great.

For desktop usecases, I'm not sure.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#263

Post by jsheard »

bcoates wrote: post #249
jsheard wrote: post #71

Earlier quoted context omitted.

> But tomorrow, will they just be included by default? That's already the way things are going due to Microsoft decreeing that Copilot+ is the future of Windows, so AMD and Intel are both putting NPUs which meet the Copilot+ performance standard into every consumer part they make going forwards to secure OEM sales.

Microsoft has declared a whole lot of things to be the future of Windows, almost all of them were quietly sidelined in a version or two. https://www.joelonsoftware.com/2002/01/06/fire-and-motion/

Yeah, but the lead times on silicon mean we're going to be stuck with Microsoft's decision for while regardless of how hard they commit to it. AMD and Intel probably already have two or three future generations of Copilot+ CPUs in the pipeline.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#264

Post by caeril »

theGnuMe wrote: post #261
david-gpu wrote: post #238

Earlier quoted context omitted.

Most of the computations are done inside NVidia proprietary libraries, not open-source CUDA. And if you saw what goes inside those libraries, I think you would agree that it is a substantial moat.

There are clean room approaches like AMDs and Scale.

Geohot has multiple (and ongoing) rants about the sheer instability of AMD RDNA3 drivers. Lisa Su engaged directly with him on this, and she didn't seem to give a shit about their problems.

AMD is not taking ML applications seriously, outside of their marketing hype.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#265

Post by cjbgkagh »

Hizonner wrote: post #239
cjbgkagh wrote: post #136

> We've tried to avoid that by making both the input matrices more square, so that tiling and reuse should be possible. While it might be possible it would not surprise me if a number of possible optimizations had not made it into Onnx. It appears that Qualcomm does not give direct access to the NPU and users are expected to use frameworks to convert models over to it, and in my experience conversion tools generally…

> While LLMs are the new hotness there are huge number of specialized tasks that small models are really useful for. Can you give some examples? Preferably examples that will run continuously enough for even a small model to stay in cache, and are valuable enough to a significant number of users to justify that cache footprint? I am not saying there aren't any, but I also honestly don't know what they are and would l…

I guess these days basically anything in ML prior to LLMs would be considered small. LLMs are rather unusual because of how large they are.

NNs can be used as a general function approximators so any function which can be approximated is a candidate for using a NN in it's place. I have a very complex trig function that produces a high dimensional smooth manifold which I know will only be used within a narrow range of inputs and I can sacrifice some accuracy for speed. My inner loops have inner loops which have inner loops with inner loops. So when you're 4+ inner loops deep the speed becomes essential. I can sweep the entire input domain to make sure the error always stays within limits.

If you're doing things such as counting instructions, intrinics, inline assembly, bit-twiddling, fast math, polynomial approximations, LUTs, fixed point math, etc. you could probably add NNs to your toolkit.

Stockfish uses a 'small' 82K parameter neural net of 3 dense integer only layers (https://news.ycombinator.com/item?id=27734517). I think Stockfish performance would be a really good candidate for testing NPUs as there is a time / accuracy tradeoff.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#266

Post by sounds »

pclmulqdq wrote: post #179
AlexandrB wrote: post #16

Earlier quoted context omitted.

> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. I have a sneaking suspicion that the real real reason for an NPU is marketing. "Oh look, NVDA is worth $3.3T - let's make sure we stick some AI stuff in our products too."

The correct way to make a true "NPU" is to 10x your memory bandwidth and feed a regular old multicore CPU with SIMD/vector instructions (and maybe a matrix multiply unit). Most of these small NPUs are actually made for CNNs and other models where "stream data through weights" applies. They have a huge speedup there. When you stream weights across data (any LLM or other large model), you are almost certain to be bound…

Apple Silicon is surprisingly a good approach here -

   * On CPU: SIMD NEON
   * On CPU: custom matrix multiply accelerator, separate from SIMD unit
   * On CPU package: NPU
   * GPU
Then they go and hide it all in proprietary undocumented features and force you to use their framework to access it :c

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#267

Post by bsmartt »

what are all these folks hoping to accomplish? By crying and starting shit about windows recall, all you did was signal to their shareholders and the financial analysts that windows recall actually substance and not just a marketing facade. Otherwise, why would all those nerds be so angry?

So microsoft takes some of the criticisms on twitter and gets them in before shipping. Free appsec, nice.

Now, microsoft doesnt care about your benchmarks, dude. Grandma isnt gonna notice these workloads finish faster on a different compiled program utilizing different chips. Her last PC was EOL'd 10 years ago, it certainly cant keep up with this new ai laptop.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#268

Post by bsmartt »

You dont seriously think MSFT expects this shit to benefit consumers do you? Their datacenters are overheating and the billing meter is still ticking while they burn, they need to figure out how to get consumers to start paying for this shit before they go broke and wall st sells them off for parts..

Either way, these are some of the first personal computers to have NPUs. They will improve. CPUs are 20 years optimized, this is literally the first try for some of these companies

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#269

Post by bsmartt »

bsmartt wrote: post #268

You dont seriously think MSFT expects this shit to benefit consumers do you? Their datacenters are overheating and the billing meter is still ticking while they burn, they need to figure out how to get consumers to start paying for this shit before they go broke and wall st sells them off for parts.. Either way, these are some of the first personal computers to have NPUs. They will improve. CPUs are 20 years optimize…

if anything this is a very promising benchmark for the new tech,. we're getting close.

so what this means if NPUs are anywhere close to CPUs in the benchmarks is that NPUs are going to blow past CPUs very soon, because CPUs dont have much more weight to shed whereas NPUs are just getting started.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#270

Post by nioj »

jcgrillo wrote: post #138
ezst wrote: post #39

Earlier quoted context omitted.

I can't wait for the LLM fad to be over so we get some sanity (and efficiency) back. I personally have no use for this extra hardware ("GenAI" doesn't help me in any way nor supports any work-related tasks). Worse, most people have no use for that (and recent surveys even show predominant hostility towards AI creep). We shouldn't be paying extra for that, it should be opt-in, and then it would become clear (by lookin…

I just got an iphone and the whole photos thing is absolutely garbage. All I wanted to do was look through my damn photos and find one I took recently but it started playing some random music and organized them in no discernible order.. like it wasn't the reverse time sorted.. Idk what kind of fucked up "creative process" came up with that bullshit but I sure wish they'd unfuck it stat. The camera is real good though…

I think the music part is related to the setting called "Show Featured Content" in the Photos app settings.
Post reply on HN