Can I run AI locally?

1. I have an M3 Ultra with 256GB of memory, but the options list only goes up to 192GB. The M3 Ultra supports up to 512GB. 2. It'd be great if I could flip this around and choose a model, and then see the performance for all the different processors. Would help making buying decisions!

Re: Can I run AI locally?

#5

Post by GrayShade » Fri, Mar 13, 2026, 4:08 PM UTC

This feels a bit pessimistic. Qwen 3.5 35B-A3B runs at 38 t/s tg with llama.cpp (mmap enabled) on my Radeon 6800 XT.

Re: Can I run AI locally?

#6

Post by phelm » Fri, Mar 13, 2026, 4:13 PM UTC

This is awesome, it would be great to cross reference some intelligence benchmarks so that I can understand the trade off between RAM consumption, token rate and how good the model is

Re: Can I run AI locally?

#7

Post by S4phyre » Fri, Mar 13, 2026, 4:16 PM UTC

Oh how cool. Always wanted to have a tool like this.

Re: Can I run AI locally?

#8

Post by embedding-shape » Fri, Mar 13, 2026, 4:21 PM UTC

John23832 wrote: post #2

RTX Pro 6000 is a glaring omission.

Yeah, that's weird, seems it has later models, and earlier, but specifically not Pro 6000? Also, based on my experience, the given numbers seems to be at least one magnitude off, which seems like a lot, when I use the approx values for a Pro 6000 (96GB VRAM + 1792 GB/s)