151–160 of 189 posts
I am considering a M5 Pro (18/20C) Macbook with 64GB of RAM, but I'm having a really hard time finding benchmarks of real world models: Could somebody please provide some tokens-per-second numbers for example for Qwen 3.6 35B/A3B, specifically for Q4 and Q6 quants?
I got qwen3.6:27B running on my 4090 (24GB) with ~128K context leveraging some of the recent turboquant/rotorquant memory optimizations for activations. Highly suggest going up to that. the q4_xl+rotorquant combo is pretty good. Some reference code if you want to throw your agent at it. https://github.com/rapatel0/rq-models
I'd rather spend thousands dollars on a Mac than subscribing API. The local model allows me to do my work any time and anywhere, without worrying about privacy leak.
Gemma4 is a huge improvement and it's fast. Qwen 3.5 really slows down my machine though. LMK if there is a better model to use for the code assist aspect- performance wise
Qwen3.5/3.6 are really prone to looping and 'overthinking'. Gemma4 doesn't seem to have the same problems.
Simple test failed: sending "1","2","3" as separate messages using an openclaw harness.
I tested a few other "follow these instructions" tests. Qwen3.5/6 were able to follow along, gemma was not able to.
In this case, picking out "semantic" css classes on single dom nodes.
Was able to run it on my 4(?) year old M2 mbp with 16GB of ram and it runs in only 100ms or so per query. Probably it can run much faster, but haven't experimented with batching etc
With tight and targeted context control, you can use extremely small models for useful things. Ideally with problems where the harness can be mostly deterministic and you have known bounds on what you're trying to do
so, interested how many people are running higher end AI models locally? Figure if I'm spending $800/month on tokens I can build a pretty beefy local machine for the cost of a few months spend - what is people's experience with say a $5k server custom built (and only for) running an AI model.
I have an M4 Macbook Air with 32Gb. These are my current results for my models: ┌──────────────────────┬───────────┬─────────────┐ │ Model │ Size │ Tokens/sec │ ├──────────────────────┼───────────┼─────────────┤ │ gemma-4-e4b-it-mlx │ ~4B (MLX) │ ~10.5 tok/s │ ├──────────────────────┼───────────┼─────────────┤ │ qwen3-8b-uncensor-v2 │ 8B │ ~6.3 tok/s │ ├──────────────────────┼───────────┼─────────────┤ │ qwen3-14b-un…
Earlier quoted context omitted.
It may surprise you but over thousands of hours I have actually gathered more than one sample. EDIT: Here's another sample for ya. I went to the store to buy mixers and while I was out Gemma 4 31b got pretty far along with reverse engineering the bluetooth protocol of a desk thermometer I have. I forgot to turn on the web search tool, so it just went at it, writing more and more specific diagnostic logging/probing to…
What is your opinion on qwen 35b MOEvs qwen 27b dense?
> The longer you let it drive without constraints, the worse the wreckage gets. The velocity makes you think you're winning right up until the moment everything collapses simultaneously. In my experience (so far), I can’t let the LLM write too much in one go. I need to test the hell out of what it gives me, and I can’t ask for too much, at one time. I tend to ask it to “flesh out” functions, where I have a signature,…
> What it gives me in Swift, most closely resembles stuff that enthusiastic newer folks would do, and want to show off. The same is true for rust-lang. Code that will immediately clone/re-allocate anything passed by reference and collect everything to the heap that is passed by `Iterator`/`IntoIterator`. It is a massive performance anti-pattern and the hallmark of somebody "struggling" with the borrow checker. Natura…
What does it do if you put "Avoid unnecessary clones, collections, and copies" in your CLAUDE.md/AGENTS.md?
> The longer you let it drive without constraints, the worse the wreckage gets. The velocity makes you think you're winning right up until the moment everything collapses simultaneously. In my experience (so far), I can’t let the LLM write too much in one go. I need to test the hell out of what it gives me, and I can’t ask for too much, at one time. I tend to ask it to “flesh out” functions, where I have a signature,…
And LLMs tend to converge on mediocrity. Which is totally fine.