Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI is driving the biggest change in laptops in decades

spectrum.ieee.org

Post reply on HN
Print view View original

191–200 of 273 posts

Re: Local AI is driving the biggest change in laptops in decades

#191

Post by nunodonato »

literalAardvark wrote: post #189
nunodonato wrote: post #165

Earlier quoted context omitted.

my laptop is 4 years old. I only have 6Gb VRam. I run, mostly, 4b and 8b models. They are extremely useful in a variety of situations. Just because you can't replicate what you do in chatgpt doesn't mean they don't have their use cases. It seems to me you know very little about what these models can do. Not to speak of trained models for specific use cases, or even smaller models like functiongemma or TTS/ASR models.…

A few examples of useful tasks would be appreciated. I do suffer from a sad lack of imagination.

I suggest taking a look at /r/localLLaMa and see all sorts of cool things people do with small models.

Re: Local AI is driving the biggest change in laptops in decades

#192

Post by reactordev »

nunodonato wrote: post #165
literalAardvark wrote: post #131

Earlier quoted context omitted.

Apple has a 10-18% market share for laptops. That's significant but it certainly isn't "most". Most laptops can run at best a 7-14b model, even if you buy one with a high spec graphics chip. These are not useful models unless you're writing spam. Most desktops have a decent amount of system memory but that can't be used for running LLMs at a useful speed, especially since the stuff you could run in 32-64GB RAM would…

my laptop is 4 years old. I only have 6Gb VRam. I run, mostly, 4b and 8b models. They are extremely useful in a variety of situations. Just because you can't replicate what you do in chatgpt doesn't mean they don't have their use cases. It seems to me you know very little about what these models can do. Not to speak of trained models for specific use cases, or even smaller models like functiongemma or TTS/ASR models.…

I’ll chime in and say I run LM Studio on my 2021 MacBook Pro M1 with no issues.

I have 16GB ram. I use unsloth quantized models like qwen3 and gpt-oss. I have some MCP servers like Context7 and Fetch that make sure the models have up to date information. I use continue.dev in VSCode or OpenCode Agent with LM Studio and write C++ code against Vulkan.

It’s more than capable. Is it fast? Not necessarily. Does it get stuck? Sometimes. Does it keep getting better? With every model release on huggingface.

Total monthly cost: $0

Re: Local AI is driving the biggest change in laptops in decades

#194

Post by oa335 »

nl wrote: post #183
m4rtink wrote: post #174

Earlier quoted context omitted.

Are you sure the subscription will still be affordable after the venture capital flood ends and the dumping stops?

100% yes. The amount of compute in the world is doubling over 2 years because of the ongoing investment in AI (!!) In some scenario where new investment stops flowing and some AI companies go bankrupt all that compute will be looking for a market. Inference providers are already profitable so with cheaper hardware it will mean even cheaper AI systems.

> Inference providers are already profitable.

That surprises me, do you remember where you learned that?

Re: Local AI is driving the biggest change in laptops in decades

#195

Post by xrd »

The takeaway from these comments are that you can really run local models if you use m-series devices from apple.

But, can you do that if you install Linux on that hardware?

I hate to admit apple hardware is incredible. But, I can't say the same about macos anymore.

Can I run Linux and reap the benefits of m-series chips with local inference?

Or, are there any alternatives where I can use llms on Linux on a laptop?

Re: Local AI is driving the biggest change in laptops in decades

#196

Post by sueders101 »

andai wrote: post #126
dangus wrote: post #114

Earlier quoted context omitted.

Then idk why they say that most laptops are bad at running LLMs, Apple has a huge marketshare in the laptop market and even their cheapest laptops are capable in that realm. And their PC competitors are more likely to be generously specced out in terms of included memory. > However, for the average laptop that’s over a year old, the number of useful AI models you can run locally on your PC is close to zero. This stra…

So I'm hearing a lot of people running LLMs on Apple hardware. But is there actually anything useful you can run? Does it run at a usable speed? And is it worth the cost? Because the last time I checked the answer to all three questions appeared to be no. Though maybe it depends on what you're doing? (Although if you're doing something simple like embeddings, then you don't need the Apple hardware in the first place.…

I've tried out gpt-oss:20b on a MacBook Air (via Ollama) with 24GB of RAM. In my experience it's output is comparable to what you'd get out of older models and the openAI benchmarks seem accurate https://openai.com/index/introducing-gpt-oss/ . Definitely a usable speed. Not instant, but ~5 tokens per second of output if I had to guess.

Re: Local AI is driving the biggest change in laptops in decades

#197

Post by AyyEye »

nl wrote: post #183
m4rtink wrote: post #174

Earlier quoted context omitted.

Are you sure the subscription will still be affordable after the venture capital flood ends and the dumping stops?

100% yes. The amount of compute in the world is doubling over 2 years because of the ongoing investment in AI (!!) In some scenario where new investment stops flowing and some AI companies go bankrupt all that compute will be looking for a market. Inference providers are already profitable so with cheaper hardware it will mean even cheaper AI systems.

You should probably disclose that you're a CTO at an AI startup, I had to click your bio to see that.

> The amount of compute in the world is doubling over 2 years because of the ongoing investment in AI (!!)

All going into the hands of a small group of people that will soon need to pay the piper.

That said, VC backed tech companies almost universally pull the rug once the money stops coming in. And historically those didn't have the trillions of dollars in future obligations that the current compute hardware oligopoly has. I can't see any universe where they don't start charging more, especially now that they've begun to make computers unaffordable for normal people.

And even past the bottom dollar cost, AI provides so many fun, new, unique ways for them to rug pull users. Maybe they start forcing users to smaller/quantized models. Maybe they start giving even the paying users ads. Maybe they start inserting propaganda/ads directly into the training data to make it more subtle. Maybe they just switch out models randomly or based on instantaneous hardware demand, giving users something even more unstable than LLMs already are. Maybe they'll charge based on semantic context (I see you're asking for help with your 2015 Ford Focus. Please subscribe to our 'Mechanic+' plan for $5/month or $25 for 24 hours). Maybe they charge more for API access. Maybe they'll charge to not train on your interactions.

I'll pass, thanks.

Re: Local AI is driving the biggest change in laptops in decades

#198

Post by anonzzzies »

andai wrote: post #126
dangus wrote: post #114

Earlier quoted context omitted.

Then idk why they say that most laptops are bad at running LLMs, Apple has a huge marketshare in the laptop market and even their cheapest laptops are capable in that realm. And their PC competitors are more likely to be generously specced out in terms of included memory. > However, for the average laptop that’s over a year old, the number of useful AI models you can run locally on your PC is close to zero. This stra…

So I'm hearing a lot of people running LLMs on Apple hardware. But is there actually anything useful you can run? Does it run at a usable speed? And is it worth the cost? Because the last time I checked the answer to all three questions appeared to be no. Though maybe it depends on what you're doing? (Although if you're doing something simple like embeddings, then you don't need the Apple hardware in the first place.…

I was sitting in an airplane next to a guy on a MacBook pro something who was coding in cursor with a local llm. We got talking and he said there are obviously differences but for his style of 'English coding' (he described basically what code to write/files to change but in english, but more sloppy than code obviously otherwise he would just code) it works really well. And indeed that's what he could demo. The model (which was the OSS gpt i believe) did pretty well in his nextjs project and fast too.

Re: Local AI is driving the biggest change in laptops in decades

#199

Post by anonzzzies »

m4rtink wrote: post #174
whazor wrote: post #115

Earlier quoted context omitted.

But economically, it is still much better to buy a lower spec't laptop and to pay a monthly subscription for AI. However, I agree with the article that people will run big LLMs on their laptop N years down the line. Especially if hardware outgrows best-in-class LLM model requirements. If a phone could run a 512GB LLM model fast, you would want it.

Are you sure the subscription will still be affordable after the venture capital flood ends and the dumping stops?

They will go down. Or the company will be gone.

Re: Local AI is driving the biggest change in laptops in decades

#200

Post by jeremyjh »

nl wrote: post #183
m4rtink wrote: post #174

Earlier quoted context omitted.

Are you sure the subscription will still be affordable after the venture capital flood ends and the dumping stops?

100% yes. The amount of compute in the world is doubling over 2 years because of the ongoing investment in AI (!!) In some scenario where new investment stops flowing and some AI companies go bankrupt all that compute will be looking for a market. Inference providers are already profitable so with cheaper hardware it will mean even cheaper AI systems.

Datacenters full of GPU hosts aren't like dark fiber - they require massive ongoing expense, so the unit economics have to work really well. It is entirely possible that some overbuilt capacity will be left idle until it is obsolete.
Post reply on HN