Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI PCs Aren't Good at AI: The CPU Beats the NPU

github.com

Post reply on HN
Print view View original

201–210 of 303 posts

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#201

Post by bzzzt »

cj wrote: post #163
Spooky23 wrote: post #159

Earlier quoted context omitted.

Microsoft needs to throw something in the gap to slow down MacBook attrition. The M processors changed the game. My teams support 250k users. I went from 50 MacBooks in 2020 to over 10,000 today. I added zero staff - we manage them like iPhones.

Rightly so. The M processor really did completely eliminate all sense of “lag” for basic computing (web browsing, restarting your computer, etc). Everything happens nearly instantly, even on the first generation M1 processor. The experience of “waiting for something to load” went away. Not to mention these machines easily last 5-10 years.

Depends on the application as well. Just try to start up Microsoft Teams.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#202

Post by im3w1l »

Spooky23 wrote: post #162
brookst wrote: post #145

Earlier quoted context omitted.

The shoehorning only works if there is buyer demand. As a company, if customers are willing to pay a premium for a NPU, or if they are unwilling to buy a product without one, it is not your place to say “hey we don’t really believe in the AI hype so we’re going to sell products people don’t want to prove a point”

Apple will have a completely AI capable product line in 18 months, with the major platforms basically done. Microsoft is built around the broken Intel tick/tick model of incremental improvement — they are stuck with OEM shitware that will take years to flush out of the channel. That means for AI, they are stuck with cloud based OpenAI, where NVIDIA has them by the balls and the hyperscalers are all fighting for GPU.…

Until AI chips become abundant, and we are not there yet, cloud AI just makes too much sense. Using a chip constantly vs using it 0.1% of the time is just so many orders of magnitude better.

Local inference does have privacy benefits. I think at the moment it might make sense to send most of queries to a beefy cloud model, and send sensitive queries to a smaller local one.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#203

Post by bugbuddy »

alisonatwork wrote: post #194
bugbuddy wrote: post #173

Earlier quoted context omitted.

The problem for Intel and AMD is they are stuck with an OS that ships with a lag-inducing Anti-malware suite. I just did a simple git log and it took 2000% longer than usual because the Antivirus was triggered to scan and run a simulation on each machine instruction and byte of data accessed. The commit log window stayed blank waiting to load long enough for me to complete another tiny project. It always ruin my day.

Pro tip: turn off malware scanning in your git repos[0]. There is also the new Dev Drive feature in Windows 11 that makes it even easier for developers (and IT admins) to set this kind of thing up via policies[1]. In companies where I worked where the IT team rolled out "security" software to the Mac-based developers, their computers were not noticeably faster than Windows PCs at all, especially given the majority of…

Unfortunately, the IT department people think they are literal GODs for knowing how to configure Domain Policies and lock down everything. They even refuse to help or even answer requests for help when there are false positives on our own software builds that we cannot unmark as false positives. These people are proactively antagonistic to productivity. Management could not careless…

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#204

Post by n8cpdx »

nxobject wrote: post #165
cj wrote: post #163

Earlier quoted context omitted.

Rightly so. The M processor really did completely eliminate all sense of “lag” for basic computing (web browsing, restarting your computer, etc). Everything happens nearly instantly, even on the first generation M1 processor. The experience of “waiting for something to load” went away. Not to mention these machines easily last 5-10 years.

As a very happy M1 Max user (should've shelled out for 64GB of RAM, though, for local LLMs!), I don't look forward to seeing how the Google Workspace/Notions/etc. of the world somehow reintroduce lag back in.

Chrome managed it. Not sure how since Edge still works reasonably well and Safari is instant to start (even faster than system settings, which is really an indictment of SwiftUI).

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#205

Post by xxs »

alisonatwork wrote: post #194
bugbuddy wrote: post #173

Earlier quoted context omitted.

The problem for Intel and AMD is they are stuck with an OS that ships with a lag-inducing Anti-malware suite. I just did a simple git log and it took 2000% longer than usual because the Antivirus was triggered to scan and run a simulation on each machine instruction and byte of data accessed. The commit log window stayed blank waiting to load long enough for me to complete another tiny project. It always ruin my day.

Pro tip: turn off malware scanning in your git repos[0]. There is also the new Dev Drive feature in Windows 11 that makes it even easier for developers (and IT admins) to set this kind of thing up via policies[1]. In companies where I worked where the IT team rolled out "security" software to the Mac-based developers, their computers were not noticeably faster than Windows PCs at all, especially given the majority of…

the short answer is that you can't without the necessary permissions, and even if you do - the next roll out will wipe out your changes.

So the pro-part of the tip does not apply.

On my own machines anti-virus is one the very first things to be removed. Most of the time I'd turn off all the swap file, yet Windows doesn't overcommit and certain applications are notorious for allocating memory w/o even using it.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#206

Post by bee_rider »

defrost wrote: post #200
djur wrote: post #199

Earlier quoted context omitted.

I'm not sure what "gothic music existed in the 1980s" is meant to indicate as a response to "goths existed in the early 2000s as a cultural archetype".

That Goths in 2000's were at best second wave nostalgia cycle of Goths from the 1980s. That people recalling Goths in that period should beware of thinking that was a source and not an echo. In 2006 Noel Fielding's Richmond Felicity Avenal was a basement dwelling leftover from many years past.

True Goth died our way before any of that. They totally sold out when the sacked Rome, the gold went to their heads and everything since then has been nostalgia.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#207

Post by bee_rider »

pclmulqdq wrote: post #179
AlexandrB wrote: post #16

Earlier quoted context omitted.

> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. I have a sneaking suspicion that the real real reason for an NPU is marketing. "Oh look, NVDA is worth $3.3T - let's make sure we stick some AI stuff in our products too."

The correct way to make a true "NPU" is to 10x your memory bandwidth and feed a regular old multicore CPU with SIMD/vector instructions (and maybe a matrix multiply unit). Most of these small NPUs are actually made for CNNs and other models where "stream data through weights" applies. They have a huge speedup there. When you stream weights across data (any LLM or other large model), you are almost certain to be bound…

I’m sure we’ll get GPNPU. Low precision matvecs could be fun to play with.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#208

Post by carlob »

defrost wrote: post #191
jjmarr wrote: post #188

Earlier quoted context omitted.

Cringe emo girls are trendy now because the nostalgia cycle is hitting the early 2000s. Your kid would be impressed if you told them you were a goth gf. It's not hard to imagine the same will happen with primitive AIs in the 40s.

Early 2000's ?? Bela Lugosi Died in 1979, and Peter Murphy was onto his next band by 1984. By 2000 Goth was fully a distant dot in the rear view mirror for the OG's In 2002, Murphy released *Dust* with Turkish-Canadian composer and producer Mercan Dede, which utilizes traditional Turkish instrumentation and songwriting, abandoning Murphy's previous pop and rock incarnations, and juxtaposing elements from progressive…

There was a submission here a few months ago about the various incarnations of goth starting from the late Roman empire.

https://www.the-hinternet.com/p/the-goths

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#209

Post by jocaal »

pjmlp wrote: post #196
Spooky23 wrote: post #159

Earlier quoted context omitted.

Microsoft needs to throw something in the gap to slow down MacBook attrition. The M processors changed the game. My teams support 250k users. I went from 50 MacBooks in 2020 to over 10,000 today. I added zero staff - we manage them like iPhones.

Microsoft has indeed a problem, however only in countries whose people can afford Apple level prices, and not everyone is a G7 citizen.

Microsoft is slowly being squeezed from both sides of the market. Chromebooks have silently become wildly popular on the low end. The only advantage I see windows have is corporate and gaming. But valve is slowly chopping away at the gaming advantage as well.

Re: AI PCs Aren't Good at AI: The CPU Beats the NPU

#210

Post by ezst »

renewiltord wrote: post #51
ezst wrote: post #39

Earlier quoted context omitted.

I can't wait for the LLM fad to be over so we get some sanity (and efficiency) back. I personally have no use for this extra hardware ("GenAI" doesn't help me in any way nor supports any work-related tasks). Worse, most people have no use for that (and recent surveys even show predominant hostility towards AI creep). We shouldn't be paying extra for that, it should be opt-in, and then it would become clear (by lookin…

I was telling someone this and they gave me link to a laptop with higher battery life and better performance than my own, but I kept explaining to them that the feature I cared most about was die size. They couldn't understand it so I just had to leave them alone. Non-technical people don't get it. Die size is what I care about. It's a critical feature and so many mainstream companies are missing out on my money beca…

I'm fine with the mockery, I genuinely hadn't realized that "wanting to pay for what one needs" was such a hot and controversial take.
Post reply on HN