[flagged]
Why your local LLM feels dumber than it is
71–80 of 233 posts
Re: Why your local LLM feels dumber than it is
#72I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.
It was actually great. I have like a non-AI box so to speak 8GB VRAM, co-incidentally from a gaming PC ...
All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period.
Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev`. while any other model previously just entirely failed to create anything usable even with actual guidance.
Since then I have actually gotten a gemma-4 12B qat 4bit quantized with a ~250MB MTP from unsloth to work with a 32k context "working" on this setup at 80-120 t/s. That's usable for private stuff on a co-incidental box!
It's still only 32k context and it's entirely dumb vs. our API paid at-work Claude Opus. But for entirely private local stuff it's totally workable without breaking the bank even after all these AI price hikes!. I bought this rig literally just for gaming a month ago.
Re: Why your local LLM feels dumber than it is
#73Earlier quoted context omitted.
i suspect ppl dropping generic "its awesome" comments are not actually using it and prbly just managed to get it running for a prompt or two.
Seems threads about local LLMs on Apple hardware feature comments listing M3/4/5 at 48GB 64GB and not 128GB. That is, users with M-series hardware that have less-than-max RAM share results whereas users with max RAM do not. Speculating (not extrapolating), maybe users with machine that have max RAM are less interested in running local LLMs and are less averse to paying services for compute? Personally, I’d love to se…
Qwen3.8-27B runs at 59.5 tok/s on my M4 Max, 40-core GPU, 128 GB
I use it occasionally for classification and other tasks but I wouldn't trust those smaller models with the real work and for larger data processing it's too slow, e.g. a dataset I wanted to classify would've taken 56 days on my laptop vs just paying the cheap Luna prices to openai and getting it done in a few hours.
Re: Why your local LLM feels dumber than it is
#74Earlier quoted context omitted.
It’s hot and also LOUD and runs the battery down quick. But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in. It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever…
> It’s hot and also LOUD and runs the battery down quick. Who'd run this on battery? Or do you mean kills the battery faster even when used while charging?
Running the laptop hot ages the battery even when you’re plugged in.
Re: Why your local LLM feels dumber than it is
#75Earlier quoted context omitted.
Seems threads about local LLMs on Apple hardware feature comments listing M3/4/5 at 48GB 64GB and not 128GB. That is, users with M-series hardware that have less-than-max RAM share results whereas users with max RAM do not. Speculating (not extrapolating), maybe users with machine that have max RAM are less interested in running local LLMs and are less averse to paying services for compute? Personally, I’d love to se…
Qwen3.8-27B runs at 59.5 tok/s on my M4 Max, 40-core GPU, 128 GB I use it occasionally for classification and other tasks but I wouldn't trust those smaller models with the real work and for larger data processing it's too slow, e.g. a dataset I wanted to classify would've taken 56 days on my laptop vs just paying the cheap Luna prices to openai and getting it done in a few hours.
Not sure I would trust Luna with that. Deepseek Pro Max and Code Mode I would be more inclined to trust.
Re: Why your local LLM feels dumber than it is
#76Earlier quoted context omitted.
My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?
It’s hot and also LOUD and runs the battery down quick. But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in. It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever…
I remember doing this with crypto mining way back in the day. That machine will die within a year.
Re: Why your local LLM feels dumber than it is
#77[dead]
Re: Why your local LLM feels dumber than it is
#78[flagged]
Re: Why your local LLM feels dumber than it is
#79[flagged]
It's like tongue-in-cheek intentional slop though.
The written text is good lol
Re: Why your local LLM feels dumber than it is
#80I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.
That's funny, I downloaded the same model on my 48GB M4 Pro and gave it a problem to solve in an existing codebase, it spun its wheels for twenty minutes and then fell over dead. This was using LMStudio and pi as a harness; I never use pi for anything else, so maybe I'm holding it wrong.