Can I run AI locally? • Hacker News Board

Re: Can I run AI locally?

#51

Post by bityard » Fri, Mar 13, 2026, 5:36 PM UTC

dekhn wrote: post #26

dgrin91 wrote: post #20

Earlier quoted context omitted.

Honestly I was surprised about this. It accurately got my GPU and specs without asking for any permissions. I didnt realize I was exposing this info.

How could it not? That information is always available to userspace.

"Available to userspace" is a much different thing than "available to every website that wants it, even in private mode".

I too was a little surprised by this. My browser (Vivladi) makes a big deal about how privacy-conscious they are, but apparently browser fingerprinting is not on their radar.

Re: Can I run AI locally?

#52

Post by relaxing » Fri, Mar 13, 2026, 5:37 PM UTC

amelius wrote: post #49

What is this S/A/B/C/etc. ranking? Is anyone else using it?

Apparently S being a level above A comes from Japanese grading. I’ve been confused by that, too.

Re: Can I run AI locally?

#53

Post by tcbrah » Fri, Mar 13, 2026, 5:40 PM UTC

tbh i stopped caring about "can i run X locally" a while ago. for anything where quality matters (scripting, code, complex reasoning) the local models are just not there yet compared to API. where local shines is specific narrow tasks - TTS, embeddings, whisper for STT, stuff like that. trying to run a 70b model at 3 tok/s on your gaming GPU when you could just hit an API for like $0.002/req feels like a weird flex IMO

Re: Can I run AI locally?

#54

Post by sdingi » Fri, Mar 13, 2026, 5:41 PM UTC

When running models on my phone - either through the web browser or via an app - is there any chance it uses the phone's NPU, or will these be GPU only?

I don't really understand how the interface to the NPU chip looks from the perspective of a non-system caller, if it exists at all. This is a Samsung device but I am wondering about the general principle.

Re: Can I run AI locally?

#55

Post by swiftcoder » Fri, Mar 13, 2026, 5:42 PM UTC

bityard wrote: post #51

dekhn wrote: post #26

Earlier quoted context omitted.

How could it not? That information is always available to userspace.

"Available to userspace" is a much different thing than "available to every website that wants it, even in private mode". I too was a little surprised by this. My browser (Vivladi) makes a big deal about how privacy-conscious they are, but apparently browser fingerprinting is not on their radar.

It's pretty hard to avoid GPU fingerprinting if you have webgl/webgpu enabled

Re: Can I run AI locally?

#56

Post by dekhn » Fri, Mar 13, 2026, 5:42 PM UTC

bityard wrote: post #51

dekhn wrote: post #26

Earlier quoted context omitted.

How could it not? That information is always available to userspace.

"Available to userspace" is a much different thing than "available to every website that wants it, even in private mode". I too was a little surprised by this. My browser (Vivladi) makes a big deal about how privacy-conscious they are, but apparently browser fingerprinting is not on their radar.

We switched to talking about llmfit in this subthread, it runs as native code.

Re: Can I run AI locally?

#57

Post by amelius » Fri, Mar 13, 2026, 5:43 PM UTC

It would be great if something like this was built into ollama, so you could easily list available models based on your current hardware setup, from the CLI.

Re: Can I run AI locally?

#58

Post by lambda » Fri, Mar 13, 2026, 5:44 PM UTC

meatmanek wrote: post #40

This seems to be estimating based on memory bandwidth / size of model, which is a really good estimate for dense models, but MoE models like GPT-OSS-20b don't involve the entire model for every token, so they can produce more tokens/second on the same hardware. GPT-OSS-20B has 3.6B active parameters, so it should perform similarly to a 3-4B dense model, while requiring enough VRAM to fit the whole 20B model. (In term…

Yeah, I looked up some models I have actually run locally on my Strix Halo laptop, and its saying I should have much lower performance than I actually have on models I've tested.

For MoE models, it should be using the active parameters in memory bandwidth computation, not the total parameters.

Re: Can I run AI locally?

#59

Post by am17an » Fri, Mar 13, 2026, 5:44 PM UTC

You can still run larger MoE models using expert weight off-loading to the CPU for token generation. They are by and large useable, I get ~50 toks/second on a kimi linear 48B (3B active) model on a potato PC + a 3090

Re: Can I run AI locally?

#60

Post by bityard » Fri, Mar 13, 2026, 5:45 PM UTC

orthoxerox wrote: post #35

For some reason it doesn't react to changing the RAM amount in the combo box at the top. If I open this on my Ryzen AI Max 395+ with 32 GB of unified memory, it thinks nothing will fit because I've set it up to reserve 512MB of RAM for the GPU.

Yeah, this site is iffy at best. I didn't even see Strix Halo on the list, but I selected 128GB and bumped up the memory bandwidth. It says gpt-oss-120b "barely runs" at ~2 t/s.

In reality, gpt-oss-120b fits great on the machine with plenty of room to spare and easily runs inference north of 50 t/s depending on context.