Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI PCs Aren't Good at AI: The CPU Beats the NPU

github.com

Post reply on HN
Print view View original

271–280 of 303 posts

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#271

Post by Miraste »

ezst wrote: post #250
ginko wrote: post #229

Earlier quoted context omitted.

The extra cost of the area spent on npu cores is pretty hard to quantify. I guess removing it would allow for higher yields and number of chips per wafer but then you’d need to set up tooling for two separate runs (one with npu and one without) Add to that that most of the cost is actually the design of the chip and it’s clear why manufacturers just always add the extra features. Maybe they could sell a chip with the…

The argument boils down to "since it's there, better to keep it because making a version without it would defeat economies of scale and not save much, if at all", and that's a sensible take… under the assumption that there's a general demand for NPUs, which I contest. In practice, everyone is paying a premium for NPUs that only a minority desires, and only a fraction of that minority essentially does "something" with…

Modern SoCs already have all kinds of features with use-cases that are few and non-essential. Granted they don't take as much space as NPUS, but manufacturers are betting that if NPUs are available, software will evolve to use them regularly. If it doesn't, they'll probably go away in a few generations. But at a minimum, Microsoft and Apple seem highly committed to using them.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#272

Post by xxs »

thesuitonym wrote: post #244
bugbuddy wrote: post #203

Earlier quoted context omitted.

Unfortunately, the IT department people think they are literal GODs for knowing how to configure Domain Policies and lock down everything. They even refuse to help or even answer requests for help when there are false positives on our own software builds that we cannot unmark as false positives. These people are proactively antagonistic to productivity. Management could not careless…

They don't think they're gods, they just think you're an idiot. This is not to say that you are, or even that they believe YOU individually are an idiot, it's just that users are idiots. There are also insurance, compliance, and other constraints that IT folks have that make them unwilling to turn off scanning for you.

they are allowed to do that for the folks that produce the goods of course, it just makes a lot harder to retain the said idiots.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#273

Post by Miraste »

AlexAndScripts wrote: post #89
jsheard wrote: post #71

Earlier quoted context omitted.

> But tomorrow, will they just be included by default? That's already the way things are going due to Microsoft decreeing that Copilot+ is the future of Windows, so AMD and Intel are both putting NPUs which meet the Copilot+ performance standard into every consumer part they make going forwards to secure OEM sales.

It almost makes me want to find some use for them on my Linux box (not that is has an NPU), but I truly can't think of anything. Too small to run a meaningful LLM, and I'd want that in bursts anyway, I hate voice controls (at least with the current tech), and Recall sounds thoroughly useless. Could you do mediocre machine translation on it, perhaps? Local github copilot? An LLM that is purely used to build an abstrac…

In short: no. Current-gen NPUs are so slow they can't do anything useful. AMD and Intel have 2nd-gen ones that came out a few weeks ago, and by spec they may able to run local translation and small LLMs (haven't seen benchmarks yet), but for now they are laptop-only.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#274

Post by NebulaTrek »

We ran qprof (a Qualcomm NPU profiler) on this benchmark. The profiling results indicate that the workload was distributed to the vector cores instead of the tensor core, which provide the vast majority of the compute power in the NPU (my back of napkin math suggests that HMX is 30x stronger than HVX).

The workload is relatively small, which results in underutilization of the hardware capacity due to the overhead associated with input/output quantization-dequantization and NCHW-NHCW mapping. Padding the weights and inputs to be a multiple of 64 would also help the performance.

Edit: Link to the profiling graph https://imgur.com/a/2OKR93e

Estimated HVX compute capability 421.43*1024/8 = 1.46TOPS in int8,

in which 4 is number of vector cores

2 is number operation per cycle

1.43GHz is HVX frequency

1024bit is vector register width

8bit is precision

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#275

Post by marginalia_nu »

consteval wrote: post #260
marginalia_nu wrote: post #240

Earlier quoted context omitted.

I don't think this is correct. The difference between well optimized code and unoptimized code on the CPU is frequently at least an order of magnitude performance. Reason it doesn't seem that way is that the CPU is so fast we often bottleneck on I/O first. However, for compute-workloads like inference, it really does matter.

While this is true, the most effective optimizations you don't do yourself. The compiler or runtime does it. They get the low-hanging fruit. You can further optimize yourself, but unless your design is fundamentally bad, you're gonna be micro-optimizing. gcc -O0 and -O2 has a HUGE performance gain. We don't really have anything to auto-magically do this for models, yet. Compilers are intimately familiar with x86.

While the compiler is decent at producing code that is good in terms of saturating the instruction pipeline, there are many things the compiler simply can't help you with.

Having cache friendly memory access patterns is perhaps the biggest one. Though automatic vectorization is also still not quite there, so in cases where there's a severe bottleneck, doing that manually may still considerably improve performance, if the workload is vectorizable.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#276

Post by touisteur »

hedgehog wrote: post #259
touisteur wrote: post #235

Earlier quoted context omitted.

SHAVE from MOVIDIUS was fun, before Intel bought them out.

Did they become un-fun? There are a bunch on the new Intel CPUs.

Most of the toolchain got hidden behind openvino and there was no hardware released for years. Keembay was 'next year' for years. I have some code for DSP using it that I can't use anymore. Has Intel actually released new shave cores, with an actual dev environment ? I'm curious.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#277

Post by david-gpu »

theGnuMe wrote: post #261
david-gpu wrote: post #238

Earlier quoted context omitted.

Most of the computations are done inside NVidia proprietary libraries, not open-source CUDA. And if you saw what goes inside those libraries, I think you would agree that it is a substantial moat.

There are clean room approaches like AMDs and Scale.

Are you suggesting that Scale can take cuDNN kernels and run them at anything resembling peak performance on AMD GPUs?

Because functional compatibility is hardly useful if the performance is not up to par, and cuDNN will run specific kernels that are particularly tuned to not only a specific model of GPU, but also to the specific inputs that the user is submitting. NVidia is doing a ton of work behind the scenes to both develop high-performance kernels for their exact architecture, but also to know which ones are best for a particular application.

This is probably the main reason why I was hesitant to join AMD a few years ago and to this day it seems like it was a good decision.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#278

Post by hedgehog »

touisteur wrote: post #276
hedgehog wrote: post #259

Earlier quoted context omitted.

Did they become un-fun? There are a bunch on the new Intel CPUs.

Most of the toolchain got hidden behind openvino and there was no hardware released for years. Keembay was 'next year' for years. I have some code for DSP using it that I can't use anymore. Has Intel actually released new shave cores, with an actual dev environment ? I'm curious.

The politics behind the software issues are complex. At least from the public presentation the new SHAVE cores are not much changed besides bigger vector units. I don't know what it would take to make a lower level SDK available again but it sure seems like it would be useful.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#279

Post by jcgrillo »

nioj wrote: post #270
jcgrillo wrote: post #138

Earlier quoted context omitted.

I just got an iphone and the whole photos thing is absolutely garbage. All I wanted to do was look through my damn photos and find one I took recently but it started playing some random music and organized them in no discernible order.. like it wasn't the reverse time sorted.. Idk what kind of fucked up "creative process" came up with that bullshit but I sure wish they'd unfuck it stat. The camera is real good though…

I think the music part is related to the setting called "Show Featured Content" in the Photos app settings.

Yeah that's how I ended up making it stop but if this is what these NPUs are being used for just.. why?

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#280

Post by NebulaTrek »

NebulaTrek wrote: post #274

We ran qprof (a Qualcomm NPU profiler) on this benchmark. The profiling results indicate that the workload was distributed to the vector cores instead of the tensor core, which provide the vast majority of the compute power in the NPU (my back of napkin math suggests that HMX is 30x stronger than HVX). The workload is relatively small, which results in underutilization of the hardware capacity due to the overhead ass…

The formula was formatted wrong, it should be 4 * 2 * 1.43 * 1024 / 8
Post reply on HN