Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

451–460 of 804 posts

Re: Local AI needs to be the norm

#451

Post by thot_experiment »

root_axis wrote: post #421
thot_experiment wrote: post #403

Earlier quoted context omitted.

Flat wrong. Q6 Gemma 31b feels a lot like opus 4.5 to me when run in a harness so it can retrieve information and ground itself. The gap is not that big for a lot of usecases. Qwen MoE is fast as fuck locally for things that are oneshottable. I have subscriptions to all the major providers right now and since Gemma 4 and Qwen 3.6 came out I haven't hit limits a single time. I'm actually super surprised by the number…

Sorry but you're just seeing what you want to see. The idea that a 31b model is anywhere even in the ballpark of something like Opus 4.5 is just absurd on its face.

False. The absolute capability is irrelevant, with the proper harness 31b is more than adequate for a very large portion of the tasks I ask AI to do. The metric isn't how good the model is at Erdos Problems, it's how reliably it can remove drudgery in my life. It just autonomously reverse engineered a bluetooth protocol with minimal intervention, it's ability to react to data and ground itself is constantly impressive to me. I do a ton of testing with these models, today I had Gemma answer a physics problem that Opus 4.7 gave up on. With a decent harness and context the set of tasks where their capabilities are both good enough is very surprising. The tasks I have that stump Gemma often also stump Opus 4.7.

Re: Local AI needs to be the norm

#452

Post by amelius »

driese wrote: post #417
emadb wrote: post #393

Earlier quoted context omitted.

Do you think small models will arrive? I mean if I need to write a web application in typescript why should I use a model that knows all the programming languages and it is able to reply to any questions about almost everything? I just a need a small performant model that knows how to write web applications in typescript. That could be very helpful and easy to run on my laptop.

For the same reason that a human who is fluent in five languages can probably express themselves better in either one compared to human that only speaks one, while also having a more nuanced understanding of general grammar. From what I know, learning on a more diverse set makes a model better overall.

This might be an interesting research question: can you train a model on many languages, and then extract a much smaller model that knows only one language without much loss of quality?

Re: Local AI needs to be the norm

#453

Post by [deleted] »

pier25 wrote: post #270
Schiendelman wrote: post #267

Earlier quoted context omitted.

Other than this recent spike, it's been trending cheaper continuously for decades. In a few years 128GB will be as affordable as 12GB (what flagship phones have now) is today.

I'm sure it will happen but I don't think it will be soon. 10 years ago I was using 16GB in my MBP and today it's 48GB. It's just a 3x increase during mostly a bonanza period.

[deleted]

Re: Local AI needs to be the norm

#454

Post by amelius »

Schiendelman wrote: post #267
pier25 wrote: post #258

Earlier quoted context omitted.

How fast do you reckon most people will be able to afford 128-256GB of RAM?

Other than this recent spike, it's been trending cheaper continuously for decades. In a few years 128GB will be as affordable as 12GB (what flagship phones have now) is today.

That "spike" could be a wall ...

Re: Local AI needs to be the norm

#455

Post by Tepix »

I'm pretty sure that AI assistants will become widespread.

I consider it to be very careless to entrust your emails, your chats, your calendar, your notes, your calls, your pictures, your contacts, your location history, your waking hours, your files, your TODO list, i.e. stuff including your health data to the for-profit AI companies. The temptation to earn money with your data is just too great, plus the risk of the data being stolen and sold illegally.

Local AI should be the default. For everone who can't do local AI, we need confidential compute. Yes, it has been hacked before. But it's making it a lot harder.

Re: Local AI needs to be the norm

#456

Post by amelius »

winocm wrote: post #247
reisse wrote: post #212

Earlier quoted context omitted.

> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…

Perhaps I am the odd one out here, but a small part of me wants to see what happens when you run a proprietary SOTA model on a laptop.

You burn your lap?

Re: Local AI needs to be the norm

#457

Post by thot_experiment »

onion2k wrote: post #431
cbg0 wrote: post #415

Earlier quoted context omitted.

Your n=1 might not be very relevant outside your personal use. In less contaminated benchmarks Gemma 4 is way below Sonnet 4.5, let alone Opus models: https://swe-rebench.com/

You do need to ask whether or not Sonnet or Opus are overkill for a lot of work though. If Gemma4 with some human effort can achieve the same result as Sonnet then it's arguably a lot more cost effective as you're paying for the person to operate each one regardless.

I 100% agree with your philosophy but I wanna note that I genuinely find Gemma 4 31b to be better than Sonnet. To be clear, this makes NO sense to me, so I'm probably just high and making stuff up or just biased by a small sample size since I don't use Sonnet that often. I find that Gemma 4 makes the sort of "dumb AI" mistakes Sonnet makes less often, especially in agentic mode. I genuinely don't know how that can be true but Sonnet feels much more like "autocomplete" and Gemma 4 feels like "some facsimile of thought".

Re: Local AI needs to be the norm

#458

Post by amelius »

thot_experiment wrote: post #451
root_axis wrote: post #421

Earlier quoted context omitted.

Sorry but you're just seeing what you want to see. The idea that a 31b model is anywhere even in the ballpark of something like Opus 4.5 is just absurd on its face.

False. The absolute capability is irrelevant, with the proper harness 31b is more than adequate for a very large portion of the tasks I ask AI to do. The metric isn't how good the model is at Erdos Problems, it's how reliably it can remove drudgery in my life. It just autonomously reverse engineered a bluetooth protocol with minimal intervention, it's ability to react to data and ground itself is constantly impressiv…

This is like saying that 640kB is enough for anybody.

Re: Local AI needs to be the norm

#459

Post by pjerem »

Tepix wrote: post #455

I'm pretty sure that AI assistants will become widespread. I consider it to be very careless to entrust your emails, your chats, your calendar, your notes, your calls, your pictures, your contacts, your location history, your waking hours, your files, your TODO list, i.e. stuff including your health data to the for-profit AI companies. The temptation to earn money with your data is just too great, plus the risk of th…

> I consider it to be very careless to entrust your emails, your chats, your calendar, your notes, your calls, your pictures, your contacts, your location history, your waking hours, your files, your TODO list, i.e. stuff including your health data to the for-profit AI companies.

Still, we all do it with Google. (I don't do it anymore but i did it for mostly two decades so I include myself)

Re: Local AI needs to be the norm

#460

Post by nicce »

Gud wrote: post #437
tzm wrote: post #408

People want local AI, but only if UX is good. Tooling/harness quality may matter as much as model quality. I think the future will probably be a hybrid of: 1. local AI for simple, private, everyday tasks 2. online AI for very hard or long tasks

The UI is already great. I can’t wait to run my models locally. The sooner I can do my shit without some American mega corp gulping down all my data, the better.

I fear that easier it gets to run models locally, more expensive all the hardware gets. So at the same time it gets further and further. You should have bought the hardware yesterday.
Post reply on HN