AI PCs Aren't Good at AI: The CPU Beats the NPU • Hacker News Board

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#241

Post by Spooky23 » Thu, Oct 17, 2024, 12:54 PM UTC

hakfoo wrote: post #184

Spooky23 wrote: post #162

Earlier quoted context omitted.

Apple will have a completely AI capable product line in 18 months, with the major platforms basically done. Microsoft is built around the broken Intel tick/tick model of incremental improvement — they are stuck with OEM shitware that will take years to flush out of the channel. That means for AI, they are stuck with cloud based OpenAI, where NVIDIA has them by the balls and the hyperscalers are all fighting for GPU.…

We're still looking for "that is useful". The stuff they've been trying to sell AI to the public with is increasingly looking as absurd as every 1978 "you'll store your recipes on the home computer" argument. AI text became a Human Centipede story: Start with a coherent 10-word sentence, let AI balloon it into five pages of flowery nonsense, send it to someone else, who has their AI smash it back down to 10 meaningfu…

The product isn’t released, so I don’t think we know what is or isn’t good.

People are clearly finding LLM tech useful, and we’re barely scratching the surface.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#242

Post by fennecfoxy » Thu, Oct 17, 2024, 1:29 PM UTC

isusmelj wrote: post #13

I think the results show that just in general the compute is not used well. That the CPU took 8.4ms and GPU took 3.2ms shows a very small gap. I'd expect more like 10x - 20x difference here. I'd assume that the onnxruntime might be the issue. I think some hardware vendors just release the compute units without shipping proper support yet. Let's see how fast that will change. Also, people often mistake the reason for…

I think it's definitely possibly now (or very soon) for an LLM to write native GPU/NPU code to get itself to run on different hardware.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#243

Post by thesuitonym » Thu, Oct 17, 2024, 1:35 PM UTC

thanksgiving wrote: post #231

morsch wrote: post #213

Earlier quoted context omitted.

It's fine. For basic computing, my M3 doesn't feel much faster than my Linux desktop that's like 8 years old. I think the standard for laptops was just really, really low.

> I think the standard for laptops was just really, really low. As someone who used windows laptops, I was amazed when I saw someone sitting next to me on a public transit subway on her MacBook Pro editing images on photoshop with just her trackpad. The standard for windows laptops used to be that low (about ten or twelve years ago?) that seeing a MacBook trackpad just woke someone is a part of my permanent memory.

I don't understand the hype around Apple trackpads. 15 years ago, sure, there was a huge gulf of difference, but today? The only difference that I can see or fee, at least between lenovo or dell and apple, is that the mac trackpad is physically larger.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#244

Post by thesuitonym » Thu, Oct 17, 2024, 1:44 PM UTC

bugbuddy wrote: post #203

alisonatwork wrote: post #194

Earlier quoted context omitted.

Pro tip: turn off malware scanning in your git repos[0]. There is also the new Dev Drive feature in Windows 11 that makes it even easier for developers (and IT admins) to set this kind of thing up via policies[1]. In companies where I worked where the IT team rolled out "security" software to the Mac-based developers, their computers were not noticeably faster than Windows PCs at all, especially given the majority of…

Unfortunately, the IT department people think they are literal GODs for knowing how to configure Domain Policies and lock down everything. They even refuse to help or even answer requests for help when there are false positives on our own software builds that we cannot unmark as false positives. These people are proactively antagonistic to productivity. Management could not careless…

They don't think they're gods, they just think you're an idiot. This is not to say that you are, or even that they believe YOU individually are an idiot, it's just that users are idiots.

There are also insurance, compliance, and other constraints that IT folks have that make them unwilling to turn off scanning for you.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#245

Post by lambda-research » Thu, Oct 17, 2024, 1:55 PM UTC

The benchmark is matrix multiplcation with the shapes `(6, 1500, 256) X (6, 256, 1500)`, which just aren't that big in the AI world. I think the gap would be larger with much larger matrices.

E.g. Llama 3.1 8B which is one of the smaller models has matrix multiplications like `(batch, 14336, 4096) x (batch, 4096, 14336)`.

I just don't think this benchmark is realistic enough.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#246

Post by monkeynotes » Thu, Oct 17, 2024, 2:01 PM UTC

eightysixfour wrote: post #15

I thought the purpose of these things was not to be fast, but to be able to run small models with very little power usage? I have a newer AMD laptop with an NPU, and my power usage doesn't change using the video effects that supposedly run on it, but goes up when using the nvidia studio effects. It seems like the NPUs are for very optimized models that do small tasks, like eye contact, background blur, autocorrect mo…

I believe that low power = cheaper tokens = more affordable and sustainable, to me this is what a consumer will benefit from overall. Power hungry GPUs seem to sit better in research, commerce, and enterprise.

The Nvidia killer would be chips and memory that are affordable enough to run a good enough model on a personal device, like a smartphone.

I think the future of this tech, if the general populace buys into LLMs being useful enough to pay a small premium for the device, is personal models that by their nature provide privacy. The amount of personal information folks unload on ChatGPT and the like is astounding. AI virtual girlfriend apps frequently get fed the most darkest kinks, vulnerable admissions, and maybe even incriminating conversations, according to Redditors that are addicted to these things. This is all given away to no-name companies that stand up apps on the app store.

Google even states that if you turn Gemini history on then they will be able to review anything you talk about.

For complex token prediction that requires a bigger model the personal could switch to consulting a cloud LLM, but privacy really needs to be ensured for consumers.

I don't believe we need cutting edge reasoning, or party trick LLMs for day to day personal assistance, chat, or information discovery.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#247

Post by coldpie » Thu, Oct 17, 2024, 2:15 PM UTC

jcgrillo wrote: post #138

ezst wrote: post #39

Earlier quoted context omitted.

I can't wait for the LLM fad to be over so we get some sanity (and efficiency) back. I personally have no use for this extra hardware ("GenAI" doesn't help me in any way nor supports any work-related tasks). Worse, most people have no use for that (and recent surveys even show predominant hostility towards AI creep). We shouldn't be paying extra for that, it should be opt-in, and then it would become clear (by lookin…

I just got an iphone and the whole photos thing is absolutely garbage. All I wanted to do was look through my damn photos and find one I took recently but it started playing some random music and organized them in no discernible order.. like it wasn't the reverse time sorted.. Idk what kind of fucked up "creative process" came up with that bullshit but I sure wish they'd unfuck it stat. The camera is real good though…

There is a chronological view tucked in there somewhere, but they really do hide it behind the other crap. Once you manage to get into it, it usually stays that way, but sometimes it kicks me back out to the random nonsense view and I have to take a few minutes to find chronological again.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#248

Post by wyldfire » Thu, Oct 17, 2024, 2:19 PM UTC

irusensei wrote: post #226

Are NPUs the VLIW of our times in terms of hype?

It's interesting that you bring that up. VLIW isn't hype in this case, it's the actual architecture of the Hexagon ISA used in these PCs. As to whether AI PCs revolutionize our lives - well, that certainly might be hype.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#249

Post by bcoates » Thu, Oct 17, 2024, 2:31 PM UTC

jsheard wrote: post #71

JohnFen wrote: post #67

Earlier quoted context omitted.

> These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. This has been my thinking. Today you have to go out of your way to buy a system with an NPU, so I don't have any. But tomorrow, will they just be included by default? That seems like a waste for those of us who aren't going to be running models. I wonder what other uses they could be put…

> But tomorrow, will they just be included by default? That's already the way things are going due to Microsoft decreeing that Copilot+ is the future of Windows, so AMD and Intel are both putting NPUs which meet the Copilot+ performance standard into every consumer part they make going forwards to secure OEM sales.

Microsoft has declared a whole lot of things to be the future of Windows, almost all of them were quietly sidelined in a version or two.

https://www.joelonsoftware.com/2002/01/06/fire-and-motion/

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#250

Post by ezst » Thu, Oct 17, 2024, 2:39 PM UTC

ginko wrote: post #229

ezst wrote: post #210

Earlier quoted context omitted.

I'm fine with the mockery, I genuinely hadn't realized that "wanting to pay for what one needs" was such a hot and controversial take.

The extra cost of the area spent on npu cores is pretty hard to quantify. I guess removing it would allow for higher yields and number of chips per wafer but then you’d need to set up tooling for two separate runs (one with npu and one without) Add to that that most of the cost is actually the design of the chip and it’s clear why manufacturers just always add the extra features. Maybe they could sell a chip with the…

The argument boils down to "since it's there, better to keep it because making a version without it would defeat economies of scale and not save much, if at all", and that's a sensible take… under the assumption that there's a general demand for NPUs, which I contest.

In practice, everyone is paying a premium for NPUs that only a minority desires, and only a fraction of that minority essentially does "something" with it.

This thread really helps to show that the use-cases are few, non-essential, and that the general application landscape hasn't adopted NPUs and has very little incentive to do so (because of the alien programming model, because of hardware compat across vendors, because of the ecosystem being a moving target with little stability in sight, and because of the high-effort/low-reward in general).

I do want to be wrong, of course. Tech generally is exciting because it offers new tools to crack old problems, opening new venues and opportunities in the process. Here it looks like we have a solution in search for a problem that was set by marketing departments.