Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can I run AI locally?

canirun.ai

Post reply on HN
Print view View original

281–290 of 382 posts

Re: Can I run AI locally?

#281

Post by novok »

steve_adams_86 wrote: post #253
tempoponet wrote: post #236

Earlier quoted context omitted.

What kind of small tasks do you find it's good at? My non-coding use of agents has been related to server admin, and my local-llm use-case is for 24/7 tasks that would be cost-prohibitive. So my best guess for this would be monitoring logs, security cameras, and general home automation tasks.

That's about it. The harness is still pretty rudimentary so I'm sure the system could be more capable, and that might reveal more interesting opportunities. I don't really know. So far I've got it orchestrating a few instances to dig through logs, local emails, git repositories, and github to figure out what I've been doing and what I need to do. Opus is waayyy better at it, but Qwen does a good enough job to actuall…

Oh man you just gave me an idea to use something like qwen 3.5 to categorize a lot of emails. You can keep the context small, do it per email and just churn through a lot of crap.

Re: Can I run AI locally?

#283

Post by kristianp »

jrmg wrote: post #29

Is there a reliable guide somewhere to setting up local AI for coding (please don’t say ‘just Google it’ - that just results in a morass of AI slop/SEO pages with out of date, non-self-consistent, incorrect or impossible instructions). I’d like to be able to use a local model (which one?) to power Copilot in vscode, and run coding agent(s) (not general purpose OpenClaw-like agents) on my M2 MacBook. I know it’ll be s…

https://github.com/ggml-org/llama.cpp/releases - has mac binaries

https://unsloth.ai/docs/models/qwen3.5 - running locally guide for the Qwen 3.5 family of models, which have a range of different sizes.

Re: Can I run AI locally?

#284

Post by girvo »

constantinum wrote: post #251
girvo wrote: post #184

Earlier quoted context omitted.

I run Qwen3.5-plus through Alibaba’s coding plan (Model Studio): incredibly cheap, pretty fast, and decent. I can’t compare it to the highest released weight one though.

I also want to try Qwen 3.5 plus. I have a doubt, I see almost same pricing for both Qwen and Claude code(the difference being the highest pro plan looks cheaper), and not for the lower plans. Am I missing something, when you say “cheaper” ??

I'm using their $3 USD (currently, it will go up in price later I believe - edit: just checked and yeah, so the $10 one) lite plan, and I'm yet to get close to hitting the request limits when I swap to it once I'm out of Claude tokens.

Re: Can I run AI locally?

#285

Post by girvo »

singpolyma3 wrote: post #214
girvo wrote: post #184

Earlier quoted context omitted.

I run Qwen3.5-plus through Alibaba’s coding plan (Model Studio): incredibly cheap, pretty fast, and decent. I can’t compare it to the highest released weight one though.

Is that https://www.alibabacloud.com/help/en/model-studio/coding-pla... ? I was a bit confused that it seems to be sized in requests not tokens

Yeah that's the one. I've not managed to get close to the limits that the cheapest plan has. Though I did get to sign up at $3 a month which has been neat, too, seems that's gone now

Re: Can I run AI locally?

#286

Post by numpad0 »

AzN1337c0d3r wrote: post #126
Bluecobra wrote: post #123

Earlier quoted context omitted.

I didn’t realize that you can get 128GB of memory in a notebook, that is impressive!

Most workstation class laptops (i.e. Lenovo P-series, Dell Precision) have 4 DIMM slots and you can get them with 256 GB (at least, before the current RAM shortages). There's also the Ryzen AI Max+ 395 that has 128GB unified in laptop form factor. Only Apple has the unique dynamic allocation though.

Intel had dynamic allocation since Intel 830(2001) for Pentium III Mobile. Everything always did, especially platforms with iGPUs like Xbox 360.

Only Apple and AMD have APUs with relatively fast iGPU that becomes relevant in large local LLM(>7b) use cases.

Re: Can I run AI locally?

#287

Post by manlymuppet »

Would be useful if comparable scores for performance are added, perhaps from arena.ai or ARC. I know scores can be imperfect, but it would be nice to be able to easily see what the best model your machine can handle is.

Re: Can I run AI locally?

#288

Post by eichin »

I'm surprised that this shows anything running usefully on my 2021-era thinkpad (with "Iris Xe"'TigerLake graphics) which inspires me to ask - are external GPUs useful for this sort of thing?

Re: Can I run AI locally?

#289

Post by aneyadeng »

radlad wrote: post #104
mark_l_watson wrote: post #78

I have spent a HUGE amount of time the last two years experimenting with local models. A few lessons learned: 1. small models like the new qwen3.5:9b can be fantastic for local tool use, information extraction, and many other embedded applications. 2. For coding tools, just use Google Antigravity and gemini-cli, or, Anthropic Claude, or... Now to be clear, I have spent perhaps 100 hours in the last year configuring l…

I'd love to know how you fit smaller models into your workflow. I have an M4 Macbook Pro w/ 128GB RAM and while I have toyed with some models via ollama, I haven't really found a nice workflow for them yet.

[flagged]

Re: Can I run AI locally?

#290

Post by rootusrootus »

dugidugout wrote: post #133
rootusrootus wrote: post #73

Earlier quoted context omitted.

Someone linked to llmfit. That would be a great tool to integrate with ollama. Just highlight the one you want and tell it to install. Quick, someone go vibe code that.

The latest level of abstraction! You just release your ideas half baked in some internet connected box and wake up with products! Yahoo! Onwards into the Gestell!

Okay, now I’m tempted to set up a bluesky account that takes requests and spits out working software.

I’m certain this has already been done. It’s too obvious, and too hilarious.

Post reply on HN