Cool website. The one that I'd really like to see there is the RTX 6000 Pro Blackwell 96 GB, though.
Can I run AI locally?
21–30 of 382 posts
Re: Can I run AI locally?
#22Raspberry pi? Say 4B with 4GB of ram.
I also want to run vision like Yocto and basic LLM with TTS/STT
Re: Can I run AI locally?
#23This lacks a whole lot of mobile GPUs. It also does not understand that you can share CPU memory with the GPU, or perform various KV cache offloading strategies to work around memory limits.
It says I have an Arc 750 with 2 GB of shared RAM, because that's the GPU that renders my browser...but I actually have an RTX1000 Ada with 6 GB of GDDR6. It's kind of like an RTX 4050 (not listed in the dropdowns) with lower thermal limits. I also have 64 GB of LPDDR5 main memory.
It works - Qwen3 Coder Next, Devstral Small, Qwen3.5 4B, and others can run locally on my laptop in near real-time. They're not quite as good as the latest models, and I've tried some bigger ones (up to 24GB, it produces tokens about half as fast as I can type...which is disappointingly slow) that are slower but smarter.
But I don't run out of tokens.
Re: Can I run AI locally?
#24[dead]
Re: Can I run AI locally?
#25[flagged]
Re: Can I run AI locally?
#26Earlier quoted context omitted.
Yes. But llmfit is far more useful as it detects your system resources.
Honestly I was surprised about this. It accurately got my GPU and specs without asking for any permissions. I didnt realize I was exposing this info.
How could it not? That information is always available to userspace.
Re: Can I run AI locally?
#27I don't see my beloved 5060ti. looks great though
Re: Can I run AI locally?
#28Having the rating of how well the model will run for you is cool. I miss to also have some rating of the model capabilities (even if this is tricky). There are way too many to choose. And just looking at the parameter number or the used memory is not always a good indication of actual performance.
Re: Can I run AI locally?
#29Is there a reliable guide somewhere to setting up local AI for coding (please don’t say ‘just Google it’ - that just results in a morass of AI slop/SEO pages with out of date, non-self-consistent, incorrect or impossible instructions).
I’d like to be able to use a local model (which one?) to power Copilot in vscode, and run coding agent(s) (not general purpose OpenClaw-like agents) on my M2 MacBook. I know it’ll be slow.
I suspect this is actually fairly easy to set up - if you know how.
Re: Can I run AI locally?
#30Earlier quoted context omitted.
Yes. But llmfit is far more useful as it detects your system resources.
Honestly I was surprised about this. It accurately got my GPU and specs without asking for any permissions. I didnt realize I was exposing this info.
Do you mean the OPs website? Mine's way off.
> Estimates based on browser APIs. Actual specs may vary