Can I run AI locally? • Hacker News Board

Re: Can I run AI locally?

#11

Post by mrdependable » Fri, Mar 13, 2026, 4:25 PM UTC

This is great, I've been trying to figure this stuff out recently.

One thing I do wonder is what sort of solutions there are for running your own model, but using it from a different machine. I don't necessarily want to run the model on the machine I'm also working from.

Re: Can I run AI locally?

#12

Post by g_br_l » Fri, Mar 13, 2026, 4:31 PM UTC

could you add raspi to the list to see which ridiculously small models it can run?

Re: Can I run AI locally?

#13

Post by deanc » Fri, Mar 13, 2026, 4:31 PM UTC

twampss wrote: post #10

Is this just llmfit but a web version of it? https://github.com/AlexsJones/llmfit

Yes. But llmfit is far more useful as it detects your system resources.

Re: Can I run AI locally?

#14

Post by vova_hn2 » Fri, Mar 13, 2026, 4:31 PM UTC

It says "RAM - unknown", but doesn't give me an option to specify how much RAM I have. Why?

Re: Can I run AI locally?

#15

Post by charcircuit » Fri, Mar 13, 2026, 4:32 PM UTC

On mobile it does not show the name of the model in favor of the other stats.

Re: Can I run AI locally?

#16

Post by debatem1 » Fri, Mar 13, 2026, 4:32 PM UTC

For me the "can run" filter says "S/A/B" but lists S, A, B, and C and the "tight fit" filter says "C/D" but lists F.

Just FYI.

Re: Can I run AI locally?

#17

Post by metalliqaz » Fri, Mar 13, 2026, 4:33 PM UTC

Hugging Face can already do this for you (with much more up-to-date list of available models). Also LM Studio. However they don't attempt to estimate tok/sec, so that's a cool feature. However I don't really trust those numbers that much because it is not incorporating information about the CPU, etc. True GPU offload isn't often possible on consumer PC hardware. Also there are different quants available that make a big difference.

Re: Can I run AI locally?

#18

Post by havaloc » Fri, Mar 13, 2026, 4:34 PM UTC

Missing the A18 Neo! :)

Re: Can I run AI locally?

#19

Post by cortesoft » Fri, Mar 13, 2026, 4:36 PM UTC

mrdependable wrote: post #11

This is great, I've been trying to figure this stuff out recently. One thing I do wonder is what sort of solutions there are for running your own model, but using it from a different machine. I don't necessarily want to run the model on the machine I'm also working from.

Ollama runs a web server that you use to interact with the models: https://docs.ollama.com/quickstart

You can also use the kubernetes operator to run them on a cluster: https://ollama-operator.ayaka.io/pages/en/

Re: Can I run AI locally?

#20

Post by dgrin91 » Fri, Mar 13, 2026, 4:39 PM UTC

deanc wrote: post #13

twampss wrote: post #10

Is this just llmfit but a web version of it? https://github.com/AlexsJones/llmfit

Yes. But llmfit is far more useful as it detects your system resources.

Honestly I was surprised about this. It accurately got my GPU and specs without asking for any permissions. I didnt realize I was exposing this info.