Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI is driving the biggest change in laptops in decades

spectrum.ieee.org

Post reply on HN
Print view View original

151–160 of 273 posts

Re: Local AI is driving the biggest change in laptops in decades

#151

Post by Zambyte »

zamadatix wrote: post #57
Zambyte wrote: post #56

Earlier quoted context omitted.

How can I observe it being loaded into CPU memory? When I run a 20gb model with ollama, htop reports 3gb of total RAM usage.

Think of it like loading a moving truck where: - The house is the disk - You are the RAM - The truck is the VRAM There won't be a single time you can observe yourself carrying the weight of everything being moved out of the house because that's not what's happening. Instead you can observe yourself taking many tiny loads until everything is finally moved, at which point you yourself should not be loaded as a result o…

My confusion was on the shuffling process happening per token. If this was happening per token, it would be effectively the same as loading the model from disk every token.

Re: Local AI is driving the biggest change in laptops in decades

#152

Post by DANmode »

dangus wrote: post #114
fancyfredbot wrote: post #111

Earlier quoted context omitted.

I think the author is aware of Apple silicon. The article mentions the fact Apple has unified memory and that this is advantageous for running LLMs.

Then idk why they say that most laptops are bad at running LLMs, Apple has a huge marketshare in the laptop market and even their cheapest laptops are capable in that realm. And their PC competitors are more likely to be generously specced out in terms of included memory. > However, for the average laptop that’s over a year old, the number of useful AI models you can run locally on your PC is close to zero. This stra…

> Apple has a huge marketshare in the laptop market

Hello, from outside of California!

Re: Local AI is driving the biggest change in laptops in decades

#153

Post by DANmode »

andai wrote: post #126
dangus wrote: post #114

Earlier quoted context omitted.

Then idk why they say that most laptops are bad at running LLMs, Apple has a huge marketshare in the laptop market and even their cheapest laptops are capable in that realm. And their PC competitors are more likely to be generously specced out in terms of included memory. > However, for the average laptop that’s over a year old, the number of useful AI models you can run locally on your PC is close to zero. This stra…

So I'm hearing a lot of people running LLMs on Apple hardware. But is there actually anything useful you can run? Does it run at a usable speed? And is it worth the cost? Because the last time I checked the answer to all three questions appeared to be no. Though maybe it depends on what you're doing? (Although if you're doing something simple like embeddings, then you don't need the Apple hardware in the first place.…

Of course it depends what you’re doing.

Do you work offline often?

Essential.

Re: Local AI is driving the biggest change in laptops in decades

#154

Post by andy99 »

TrackerFF wrote: post #85

With the wild ram prices, which btw are probably going to last out 2026, I expect 8 GB ram to be the new standard going on forward. 32 GB ram will be for enthusiasts with deep pockets, and professionals. Anything over that, exclusively professionals. The conspiracy theorist inside me is telling me that big AI companies like OpenAI would rather see that people are using their puny laptops as terminals / shells only, t…

  The conspiracy theorist inside me is telling me that big AI companies...
I don’t believe in conspiracies but I do believe in incentives sometimes lining up. Now that there is a RAM heavy cloud application, cloud providers are suddenly in direct competition with consumers for scarce resources, with the winner being able to control where people run their models.

Re: Local AI is driving the biggest change in laptops in decades

#155

Post by ajb »

aappleby wrote: post #3

I predict we will see compute-in-flash before we see cheap laptops with 128+ gigs of ram.

The thing that is supposed to happen next is high-bandwidth flash. In theory, it could allow laptops to run the larger models without being extortionately costly, by loading directly from flash into the GPU (not by executing in flash) But I haven't seen figures of the actual bandwidth yet, and no doubt to start with it will be expensive. The underlying technology of flash has much higher read latency than dram, so it's not really clear (to me, at least) if they can deliver the speeds needed to remove the need to cache in VRAM just by increasing parallelism.

Re: Local AI is driving the biggest change in laptops in decades

#156

Post by ip26 »

bad_haircut72 wrote: post #144

My recent shower thought was the idea that Moores law hasnt slowed at all, we just went multi-core. Its crazy that the intel folks were so interested in optimizing for single thread CPU design they completely misunderstood where the best effort would be spent - if I had been around back then (speaking as an Elixir dev) I would have been way more interested in having 500 theead CPUs than getting down to nanometer scal…

Before LLMs, the use of parallelism on your typical laptop was limited to application level parallelism, e.g. one thread for Outlook and one for each tab in Chrome.

Re: Local AI is driving the biggest change in laptops in decades

#158

Post by Groxx »

re NPUs: they've been a marketing thing for years now, but I really have no idea how many of them are actually used when you run [whatever]. particularly after a year or two of software updates.

anyone have numbers? are they just an added expense that is supported for first party stuff for 6 months before they need a bigger model, or do they have staying power? clearly they are capable of being used to save power, but does anything do that in practice, in consumer hardware?

Re: Local AI is driving the biggest change in laptops in decades

#159

Post by cramcgrab »

seunosewa wrote: post #65

"How many TOPS do you need to run state-of-the-art models with hundreds of millions of parameters? No one knows exactly." What's he talking about? It's trivial to calculate that.

It’s trivial to ask an AI to answer that. Well, I guess we know it’s not an AI generated article!

Re: Local AI is driving the biggest change in laptops in decades

#160

Post by jki275 »

azuanrb wrote: post #119
jwr wrote: post #96

The author seems unaware of how well recent Apple laptops run LLMs. This is puzzling and puts into question the validity of anything in this article.

You still need ridiculously high spec hardware, and at Apple’s prices, that isn’t cheap. Even if you can afford it (most won't), the local models you can run are still limited and they still underperform. It’s much cheaper to pay for a cloud solution and get significantly better result. In my opinion, the article is right. We need a better way to run LLMs locally.

I bought my M1 Max w/ 64gb of ram used. It's not that expensive.

Yes, the models it can run do not perform like chatgpt or claude 4.5, but they're still very useful.

Post reply on HN