Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can I run AI locally?

canirun.ai

Post reply on HN
Print view View original

121–130 of 382 posts

Re: Can I run AI locally?

#121

Post by ProllyInfamous »

mmaunder wrote: post #117

OP can you please make it not as dark and slightly larger. Super useful otherwise. Qwen 3.5 9B is going to get a lot of love out of this.

I'm not usually one to whine, but agreed; additionally, add contrast to the modifiers (e.g. processor select). First thing I did when I visited was scale the website to 150%

Super impressive comparisons, and correlates with my perception having three seperate generations of GPU (from your list pulldown). Thanks for including the "old AMD" Polaris chipsets, which are actually still much faster than lower-spec Apple silicon. I have Ollama3.1 on a VEGA64 and it really is twice as fast as an M2Pro...

----

For anybody that thinks installing a local LLM is complicated: it's not (so long as you have more than one computer, don't tinker on your primary workhorse). I am a blue collar electrician (admittedly: geeky); no more difficult than installing linux. I used an online LLM to help me install both =D

Re: Can I run AI locally?

#122

Post by downrightmike »

mopierotti wrote: post #95

This (+ llmfit) are great attempts, but I've been generally frustrated by how it feels so hard to find any sort of guidance about what I would expect to be the most straightforward/common question: "What is the highest-quality model that I can run on my hardware, with tok/s greater than , and context limit greater than " (My personal approach has just devolved into guess-and-check, which is time consuming.) When usin…

LLMs are just special purpose calculators, as opposed to normal calculators which just do numbers and MUST be accurate. There aren't very good ways of knowing what you want because the people making the models can't read your mind and have different goals

Re: Can I run AI locally?

#123

Post by Bluecobra »

radlad wrote: post #104
mark_l_watson wrote: post #78

I have spent a HUGE amount of time the last two years experimenting with local models. A few lessons learned: 1. small models like the new qwen3.5:9b can be fantastic for local tool use, information extraction, and many other embedded applications. 2. For coding tools, just use Google Antigravity and gemini-cli, or, Anthropic Claude, or... Now to be clear, I have spent perhaps 100 hours in the last year configuring l…

I'd love to know how you fit smaller models into your workflow. I have an M4 Macbook Pro w/ 128GB RAM and while I have toyed with some models via ollama, I haven't really found a nice workflow for them yet.

I didn’t realize that you can get 128GB of memory in a notebook, that is impressive!

Re: Can I run AI locally?

#126

Post by AzN1337c0d3r »

Bluecobra wrote: post #123
radlad wrote: post #104

Earlier quoted context omitted.

I'd love to know how you fit smaller models into your workflow. I have an M4 Macbook Pro w/ 128GB RAM and while I have toyed with some models via ollama, I haven't really found a nice workflow for them yet.

I didn’t realize that you can get 128GB of memory in a notebook, that is impressive!

Most workstation class laptops (i.e. Lenovo P-series, Dell Precision) have 4 DIMM slots and you can get them with 256 GB (at least, before the current RAM shortages).

There's also the Ryzen AI Max+ 395 that has 128GB unified in laptop form factor.

Only Apple has the unique dynamic allocation though.

Re: Can I run AI locally?

#127

Post by kuon »

I have amd 9700 and it is not listed while it is great llm hardware because it has 32Gb for a reasonable price. I tried doing "custom" but it didn't seem to work.

The tool is very nice though.

Re: Can I run AI locally?

#128

Post by hatthew »

tcbrah wrote: post #53

tbh i stopped caring about "can i run X locally" a while ago. for anything where quality matters (scripting, code, complex reasoning) the local models are just not there yet compared to API. where local shines is specific narrow tasks - TTS, embeddings, whisper for STT, stuff like that. trying to run a 70b model at 3 tok/s on your gaming GPU when you could just hit an API for like $0.002/req feels like a weird flex I…

For me and probably many other people, local has nothing to do with cost and everything to do with privacy

Re: Can I run AI locally?

#129

Post by fzzzy »

nine_k wrote: post #83
mark_l_watson wrote: post #78

I have spent a HUGE amount of time the last two years experimenting with local models. A few lessons learned: 1. small models like the new qwen3.5:9b can be fantastic for local tool use, information extraction, and many other embedded applications. 2. For coding tools, just use Google Antigravity and gemini-cli, or, Anthropic Claude, or... Now to be clear, I have spent perhaps 100 hours in the last year configuring l…

What kind of hardware did you use? I suppose that a 8GB gaming GPU and a Mac Pro with 512 GB unified RAM give quite different results, both formally being local.

A Mac Pro with 512 gb unified ram does not exist.
Post reply on HN