Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can I run AI locally?

canirun.ai

Post reply on HN
Print view View original

71–80 of 382 posts

Re: Can I run AI locally?

#71

Post by AstroBen »

jrmg wrote: post #66
AstroBen wrote: post #36

Earlier quoted context omitted.

Ollama or LM Studio are very simple to setup. You're probably not going to get anything working well as an agent on an M2 MacBook, but smaller models do surprisingly well for focused autocomplete. Maybe the Qwen3.5 9B model would run decently on your system?

Right - setting up LM studio is not hard. But how do I connect LM Studio to Copilot, or set up an agent?

It looks like Copilot has direct support for Ollama if you're willing to set that up: https://docs.ollama.com/integrations/vscode

For LM Studio under server settings you can start a local server that has an OpenAI-compatible API. You'd need to point Copilot to that. I don't use Copilot so not sure of the exact steps there

Re: Can I run AI locally?

#72

Post by NortySpock »

jrmg wrote: post #66
AstroBen wrote: post #36

Earlier quoted context omitted.

Ollama or LM Studio are very simple to setup. You're probably not going to get anything working well as an agent on an M2 MacBook, but smaller models do surprisingly well for focused autocomplete. Maybe the Qwen3.5 9B model would run decently on your system?

Right - setting up LM studio is not hard. But how do I connect LM Studio to Copilot, or set up an agent?

I tried the Zed editor and it picked up Ollama with almost no fiddling, so that has allowed me to run Qwen3.5:9B just by tweaking the ollama settings (which had a few dumb defaults, I thought, like assuming I wanted to run 3 LLMs in parallel, initially disabling Flash Attention, and having a very short context window...).

Having a second pair of "eyes" to read a log error and dig into relevant code is super handy for getting ideas flowing.

Re: Can I run AI locally?

#73

Post by rootusrootus »

amelius wrote: post #57

It would be great if something like this was built into ollama, so you could easily list available models based on your current hardware setup, from the CLI.

Someone linked to llmfit. That would be a great tool to integrate with ollama. Just highlight the one you want and tell it to install.

Quick, someone go vibe code that.

Re: Can I run AI locally?

#74

Post by AstroBen »

brcmthrowaway wrote: post #68
jrmg wrote: post #66

Earlier quoted context omitted.

Right - setting up LM studio is not hard. But how do I connect LM Studio to Copilot, or set up an agent?

Basically LM Studio has a server that serves models over HTTP (localhost). Configure/enable the server and connect OpenCode to it. Try this article https://advanced-stack.com/fields-notes/qwen35-opencode-lm-s... I'm looking for an alternative to OpenCode though, I can barely see the UI.

Codex also supports configuring an alternative API for the model, you could try that: https://unsloth.ai/docs/basics/codex#openai-codex-cli-tutori...

Re: Can I run AI locally?

#75

Post by cafed00d »

Open with multiple browsers (safari vs chrome) to get more "accurate + glanceable" rankings.

Its using WebGPU as a proxy to estimate system resource. Chrome tends to leverage as much resources (Compute + Memory) as the OS makes available. Safari tends to be more efficient.

Maybe this was obvious to everyone else. But its worth re-iterating for those of us skimmers of HN :)

Re: Can I run AI locally?

#77

Post by johnisgood »

dgrin91 wrote: post #20
deanc wrote: post #13

Earlier quoted context omitted.

Yes. But llmfit is far more useful as it detects your system resources.

Honestly I was surprised about this. It accurately got my GPU and specs without asking for any permissions. I didnt realize I was exposing this info.

Why were you surprised?

You can check out here how it does that: https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/s...

To detect NVIDIA GPUs, for example: https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/s...

In this case it just runs the command "nvidia-smi".

Note: llmfit is not web-based.

Re: Can I run AI locally?

#78

Post by mark_l_watson »

I have spent a HUGE amount of time the last two years experimenting with local models.

A few lessons learned:

1. small models like the new qwen3.5:9b can be fantastic for local tool use, information extraction, and many other embedded applications.

2. For coding tools, just use Google Antigravity and gemini-cli, or, Anthropic Claude, or...

Now to be clear, I have spent perhaps 100 hours in the last year configuring local models for coding using Emacs, Claude Code (configured for local), etc. However, I am retired and this time was a lot of fun for me: lot's of efforts trying to maximize local only results. I don't recommend it for others.

I do recommend getting very good at using embedded local models in small practical applications. Sweet spot.

Re: Can I run AI locally?

#79

Post by andy_ppp »

Is it correct that there's zero improvement in performance between M4 (+Pro/Max) and M5 (+Pro/Max) the data looks identical. Also the memory does not seem to improve performance on larger models when I thought it would have?

Love the idea though!

EDIT: Okay the whole thing is nonsense and just some rough guesswork or asking an LLM to estimate the values. You should have real data (I'm sure people here can help) and put ESTIMATE next to any of the combinations you are guessing.

Post reply on HN