I wish creators of local model inference tools (LM Studio, Ollama etc.) would release these numbers publicly, because you can be sure they are sitting on a large dataset of real-world performance.
Can I run AI locally?
141–150 of 382 posts
Re: Can I run AI locally?
#142Chrome runs Gemini Nano if you flip a few feature flags on [1].
The model is not great, but it was the "least amount of setup" LLM I could run on someone else's machine.
Including structured output, but has a tiny context window I could use.
Re: Can I run AI locally?
#143This shows no models work with my hardware but that’s furthest from the truth as I’m running Qwen3.5… This isn’t nearly complete.
Well… don’t keep us guessing -what hardware? And which size qwen3.5?
Re: Can I run AI locally?
#144I have spent a HUGE amount of time the last two years experimenting with local models. A few lessons learned: 1. small models like the new qwen3.5:9b can be fantastic for local tool use, information extraction, and many other embedded applications. 2. For coding tools, just use Google Antigravity and gemini-cli, or, Anthropic Claude, or... Now to be clear, I have spent perhaps 100 hours in the last year configuring l…
I'd love to know how you fit smaller models into your workflow. I have an M4 Macbook Pro w/ 128GB RAM and while I have toyed with some models via ollama, I haven't really found a nice workflow for them yet.
Shouldn't we prioritize large scale open weights and open source cloud infra?
An OpenRunPod with decent usage might encourage more non-leading labs to dump foundation models into the commons. We just need infra to run it. Distilling them down to desktop is a fool's errand. They're meant to run on DC compute.
I'm fine with running everything in the cloud as long as we own the software infra and the weights.
This is conceivably the only way we could catch up to Claude Code is to have the Chinese start releasing their best coding models and for them to get significant traction with companies calling out to hosted versions. Otherwise, we're going to be stuck in a take off scenario with no bridge.
Re: Can I run AI locally?
#145This (+ llmfit) are great attempts, but I've been generally frustrated by how it feels so hard to find any sort of guidance about what I would expect to be the most straightforward/common question: "What is the highest-quality model that I can run on my hardware, with tok/s greater than , and context limit greater than " (My personal approach has just devolved into guess-and-check, which is time consuming.) When usin…
What is the $/Mtok that would make you choose your time vs savings of running stuff locally?
Just to be clear, it may sound like a snarky comment but I'm really curious from you or others how do you see it. I mean there are some batches long running tasks where ignoring electricity it's kind of free but usually local generation is slower (and worse quality) and we all kind of want some stuff to get done.
Or is it not about the cost at all, just about not pushing your data into the clouds.
Re: Can I run AI locally?
#146Earlier quoted context omitted.
> Is it correct that there's zero improvement in performance between M4 (+Pro/Max) and M5 (+Pro/Max) Preliminary testing did not come to that conclusion. > Apple’s New M5 Max Changes the Local AI Story https://www.youtube.com/watch?v=XGe7ldwFLSE
From the video: 4.4k is "almost" 4x times 1.8k because 4.4k has "number 4" in the beginning, and the other one - number 1. For the lazy: that's less then 3x: 1.8 * 3 = 5.4
It’s not even the largest part, just prefill so I think maybe M5 Max is 30% faster overall. Still pretty good I think but the 4x nonsense is just marketing!
Re: Can I run AI locally?
#147Earlier quoted context omitted.
Most workstation class laptops (i.e. Lenovo P-series, Dell Precision) have 4 DIMM slots and you can get them with 256 GB (at least, before the current RAM shortages). There's also the Ryzen AI Max+ 395 that has 128GB unified in laptop form factor. Only Apple has the unique dynamic allocation though.
> Only Apple has the unique dynamic allocation though. What do you mean? On Linux I can dynamically allocate memory between CPU and GPU. Just have to set a few kernel parameters to set the max allowable allocation to the GPU, and set the BIOS to the minimum amount of dedicated graphics memory.
Maybe things have changed but the last time I looked at this, it was only max 96GB to the GPU. And it isn't dynamic in the sense you still have to tweak the kernel parameters, which require a reboot.
Apple has none of this.
Re: Can I run AI locally?
#148Our work at DoShare is a lot of this stuff we've been on it for 2 years
Re: Can I run AI locally?
#149Earlier quoted context omitted.
I didn’t realize that you can get 128GB of memory in a notebook, that is impressive!
I've got a 128 GiB unified memory Ryzen Ai Max+ 395 (aka Strix Halo) laptop. Trying to run LLM models somehow makes 128 GiB of memory feel incredibly tight. I'm frequently getting OOMs when I'm running models that are pushing the limits of what this can fit, I need to leave more memory free for system memory than I was expecting. I was expecting to be able to run models of up to ~100 GiB quantized, leaving 28 GiB for…
There's been some very recent ongoing work in some local AI frameworks on enabling mmap by default, which can potentially obviate some RAM-driven limitations especially for sparse MoE models. Running with mmap and too little RAM will then still come with severe slowdowns since read-only model parameters will have to be shuttled in from storage as they're needed, but for hardware with fast enough storage and especially for models that "almost" fit in the RAM filesystem cache, this can be a huge unblock at negligible cost. Especially if it potentially enables further unblocks via adding extra swap for K-V cache and long context.
Re: Can I run AI locally?
#150I have been trying to run Qwen Coder models (8B at 4bit) on my M3 Pro 18GB behind Ollama and connecting codex CLI to it. The tool usage seems practically zero, like it returns the tool call in text JSON and codex CLI doesn’t run the tool (just displays the tool call in text). Has anyone succeeded in doing something like this? What am I missing?