Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can I run AI locally?

canirun.ai

Post reply on HN
Print view View original

361–370 of 382 posts

Re: Can I run AI locally?

#361

Post by boppo1 »

sdrinf wrote: post #151
mark_l_watson wrote: post #78

I have spent a HUGE amount of time the last two years experimenting with local models. A few lessons learned: 1. small models like the new qwen3.5:9b can be fantastic for local tool use, information extraction, and many other embedded applications. 2. For coding tools, just use Google Antigravity and gemini-cli, or, Anthropic Claude, or... Now to be clear, I have spent perhaps 100 hours in the last year configuring l…

Just want to echo the recommendation for qwen3.5:9b. This is a smol, thinking, agentic tool-using, text-image multimodal creature, with very good internal chains of thought. CoT can be sometimes excessive, but it leads to very stable decision-making process, even across very large contexts -something we haven't seen models of this size before. What's also new here, is VRAM-context size trade-off: for 25% of it's atte…

Is qwen 3.5 any good for chatting? I use chatgpt for 'light therapy' (basically sounding out confusing social situations my friends don't want to walk me through) and it's honestly been amazing. But I would rather not give all that to openai.

Re: Can I run AI locally?

#362

Post by dyauspitr »

never_inline wrote: post #344
dyauspitr wrote: post #292

Earlier quoted context omitted.

For learning and general searching I find ChatGPT to be the best. Nano Banana Pro for anything image and video related. Grok Imagine for pretty decent porn generation.

Didn't want to hear the grok thing from handle named "dyauspitr". My day is ruined.

I would imagine the Dyaus was pretty randy if Zeus is anything to go by.

Re: Can I run AI locally?

#364

Post by dexterlagan »

Same as top comment, have spent a lot of time on local models. IMHO, qwen3.5 is the very first model that is actually usable for serious work, ever - and I've tried them all. The 35B 3B is very smart. It understands things no other local model I've ever used does, it's that good. The 9B runs on my slow Mac, and it's also very 'smart'. I can say with confidence that 2026 is the year of the local model, at last.

Re: Can I run AI locally?

#365

Post by Western0 »

I tried using agents on a small Orange Pi and other small machines, and I’ve come to the conclusion that, unfortunately, it’s not feasible to do this in a practical way. Of course, I started writing an agent that could run in such an environment (long timeouts, retries, etc.), but it’s a real pain. For this to make sense, you need much more powerful hardware (a 5-year-old Mac mini is fine); the other issue is power consumption. Unfortunately, until you can run Mavericks 2 on your own hardware, it’s pretty expensive.

Re: Can I run AI locally?

#367

Post by Western0 »

mark_l_watson wrote: post #78

I have spent a HUGE amount of time the last two years experimenting with local models. A few lessons learned: 1. small models like the new qwen3.5:9b can be fantastic for local tool use, information extraction, and many other embedded applications. 2. For coding tools, just use Google Antigravity and gemini-cli, or, Anthropic Claude, or... Now to be clear, I have spent perhaps 100 hours in the last year configuring l…

I only put this https://unsloth.ai/docs/models/qwen3.5 (please look at old gemma 3n 4eb, and small IBM granite)

Re: Can I run AI locally?

#368

Post by rurban »

No, not yet. Up to a single H100 there is no single local model, which doesn't make your code worse. (excluding trivial stuff like ruby, python, typescript). Implement features, fix bugs.

Right now we started experimenting with 2 H100's, 160GB models. But even a single one is wide out of anyone others league.

Re: Can I run AI locally?

#370

Post by [deleted] »

spudlyo wrote: post #63
dgrin91 wrote: post #20

Earlier quoted context omitted.

Honestly I was surprised about this. It accurately got my GPU and specs without asking for any permissions. I didnt realize I was exposing this info.

I run LibreWolf, which is configured to ask me before a site can use WebGL, which is commonly used for fingerprinting. I got the popup on this site, so I assume that's how they're doing it.

[deleted]
Post reply on HN