Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI is driving the biggest change in laptops in decades

spectrum.ieee.org

Post reply on HN
Print view View original

131–140 of 273 posts

Re: Local AI is driving the biggest change in laptops in decades

#131

Post by literalAardvark »

dangus wrote: post #114
fancyfredbot wrote: post #111

Earlier quoted context omitted.

I think the author is aware of Apple silicon. The article mentions the fact Apple has unified memory and that this is advantageous for running LLMs.

Then idk why they say that most laptops are bad at running LLMs, Apple has a huge marketshare in the laptop market and even their cheapest laptops are capable in that realm. And their PC competitors are more likely to be generously specced out in terms of included memory. > However, for the average laptop that’s over a year old, the number of useful AI models you can run locally on your PC is close to zero. This stra…

Apple has a 10-18% market share for laptops. That's significant but it certainly isn't "most".

Most laptops can run at best a 7-14b model, even if you buy one with a high spec graphics chip. These are not useful models unless you're writing spam.

Most desktops have a decent amount of system memory but that can't be used for running LLMs at a useful speed, especially since the stuff you could run in 32-64GB RAM would need lots of interaction and hand holding.

And that's for the easy part, inference. Training is much more expensive.

Re: Local AI is driving the biggest change in laptops in decades

#133

Post by zozbot234 »

RobotToaster wrote: post #107
seunosewa wrote: post #65

"How many TOPS do you need to run state-of-the-art models with hundreds of millions of parameters? No one knows exactly." What's he talking about? It's trivial to calculate that.

Isn't the ability to run it more dependant on (V)RAM? With TOPS just dictating the speed at which it runs?

Strictly speaking, you don't need that much VRAM or even plain old RAM - just enough to store your context and model activations. It's just that as you run with less and less (V)RAM you'll start to bottleneck on things like SSD transfer bandwidth and your inference speed goes down to a crawl. But even that may or may not be an issue depending on your exact requirements: perhaps you don't need your answer instantly and can wait while it gets computed in the background. Or maybe you're running with the latest PCIe 5 storage which overall gives you comparable bandwidth to something like DDR3/DDR4 memory.

Re: Local AI is driving the biggest change in laptops in decades

#134

Post by joshred »

ignoramous wrote: post #121
whazor wrote: post #115

Earlier quoted context omitted.

But economically, it is still much better to buy a lower spec't laptop and to pay a monthly subscription for AI. However, I agree with the article that people will run big LLMs on their laptop N years down the line. Especially if hardware outgrows best-in-class LLM model requirements. If a phone could run a 512GB LLM model fast, you would want it.

> economically, it is still much better to buy a lower spec't laptop and to pay a monthly subscription for AI Uber is economical, too; but folks prefer to own cars, sometimes multiple. And how there's market for all kinds of vanity cars, fast sportscars, expensive supercars... I imagine PCs & Laptops will have such a market, too: In probably less than a decade, may be a £20k laptop running a 671b+ LLM locally will be…

Paying $30-$70/day to commute is economical?

Re: Local AI is driving the biggest change in laptops in decades

#135

Post by noman-land »

seanmcdirmid wrote: post #8

I’ve been running LLMs on my laptop (M3 Max 64GB) for a year now and I think they are ready, especially with how good mid sized models are getting. I’m pretty sure unified memory and energy efficient GPUs will be more than just a thing on Apple laptops in the next few years.

You doing code completion and agentic stuff successfully with local models? Got any tips? I've been out of the game for [checks watch] a few months and am behind on the latest. Is Cline the move?

Re: Local AI is driving the biggest change in laptops in decades

#136

Post by tossandthrow »

almosthere wrote: post #128
azuanrb wrote: post #119

Earlier quoted context omitted.

You still need ridiculously high spec hardware, and at Apple’s prices, that isn’t cheap. Even if you can afford it (most won't), the local models you can run are still limited and they still underperform. It’s much cheaper to pay for a cloud solution and get significantly better result. In my opinion, the article is right. We need a better way to run LLMs locally.

749 for an M4 air at Amazon right now

Try running anything interesting on these 8gb of ram.

You need 96gb or 128gb to do non trivial things. That is not yet 749 usd

Re: Local AI is driving the biggest change in laptops in decades

#137

Post by NitpickLawyer »

RobotToaster wrote: post #107
seunosewa wrote: post #65

"How many TOPS do you need to run state-of-the-art models with hundreds of millions of parameters? No one knows exactly." What's he talking about? It's trivial to calculate that.

Isn't the ability to run it more dependant on (V)RAM? With TOPS just dictating the speed at which it runs?

A good rule of thumb is that PP (Prompt Processing) is compute bound while TG (Token Generation) is (V)RAM speed bound.

Re: Local AI is driving the biggest change in laptops in decades

#138

Post by gamblor956 »

The "AI laptop" boom is already fading. It turns out that LLMs, local or otherwise, just aren't very useful.

Like Big Data, LLMs are useful in a small niche of areas, like poorly summarizing meeting notes, or grammar check at a middle-school level.

On LLMs for coding tasks: I asked a programmer why they loved Claude and he showed me the output. Twenty years ago, that kind of code would have gotten someone PIP'd. Today it's considered better than most junior programmers...which is a sign of how far programming standards have fallen, and explains why most programs and apps are such buggy pieces of sh$t these days.

Re: Local AI is driving the biggest change in laptops in decades

#139

Post by subjectsigma »

ignoramous wrote: post #121
whazor wrote: post #115

Earlier quoted context omitted.

But economically, it is still much better to buy a lower spec't laptop and to pay a monthly subscription for AI. However, I agree with the article that people will run big LLMs on their laptop N years down the line. Especially if hardware outgrows best-in-class LLM model requirements. If a phone could run a 512GB LLM model fast, you would want it.

> economically, it is still much better to buy a lower spec't laptop and to pay a monthly subscription for AI Uber is economical, too; but folks prefer to own cars, sometimes multiple. And how there's market for all kinds of vanity cars, fast sportscars, expensive supercars... I imagine PCs & Laptops will have such a market, too: In probably less than a decade, may be a £20k laptop running a 671b+ LLM locally will be…

> Uber is economical, too

One time I took an Uber to work because my car broke down and was in the shop and the Uber driver (somewhat pointedly) made a comment that I must be really rich to commute to work via Uber because Ubers are so expensive

Re: Local AI is driving the biggest change in laptops in decades

#140

Post by zozbot234 »

re-thc wrote: post #117
seanmcdirmid wrote: post #90

Earlier quoted context omitted.

We’ve been through multiple cycles of scarcity/surplus DRAM cycles in the last couple of decades. Why do we think it will be different now?

> Why do we think it will be different now? Margins. AI usage can pay a lot more. Even if they sell less than can still be more profitable. In the past there wasn’t a high margin usage. Servers didn’t charge such a high premium.

High margins are exactly what should create a strong incentive to build more capacity. But that dynamic has been tamped down so far because we're all scared of a possible AI bubble that might pop at any moment.
Post reply on HN