Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why your local LLM feels dumber than it is

forum.level1techs.com

Post reply on HN
Print view View original

21–30 of 233 posts

Re: Why your local LLM feels dumber than it is

#21

Post by b112 »

kangalioo wrote: post #19
JacobJack wrote: post #17

> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts. Genuine question : is there something fundamentally wrong with Ollama ? I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly). I thought the main advantage of VLLM was better concurrency management (better batching). But if the quality of the inte…

From what I've heard, Ollama has a bad reputation because it's a thin wrapper around llama.cpp without attributing it properly, thereby stealing recognition from the maintainers doing most of the work

It seems, and that seems is entirely my unvalidated impression, that Ollama lags in features, as they're integrating after the fact those changes. But (seriously) an LLM told me that, when some aspects of MoE models were better supported with the latest llama.

And it did in that case make a significant difference.

Re: Why your local LLM feels dumber than it is

#23

Post by dominotw »

chorlton2080 wrote: post #8
StarlaAtNight wrote: post #5

Earlier quoted context omitted.

how quick does it respond? what are specs of your laptop?

Does it need to respond fast? For important applications, I'm sure we'd all be fine waiting 20 minutes for a high quality, usable answer. Or is it the need for interative refinements that make speed relevant?

if you are so sure about what the final shape of your output is then its prbly not a common use of ai

Re: Why your local LLM feels dumber than it is

#24

Post by MrScruff »

alexchantavy wrote: post #9
jonplackett wrote: post #2

I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.

How many tok/s are you getting? What gen mbp?

I get around 20 tok/s, 4 bit quant, MTP, 4 bit KV cache quantisation. On an M4 Pro 48Gb.

Re: Why your local LLM feels dumber than it is

#26

Post by b112 »

alexpotato wrote: post #16
Gareth321 wrote: post #12

Earlier quoted context omitted.

I tried it on my M1 MacBook Pro. It's slow but surprisingly smart as a general purpose LLM. Maybe GPT-5.3 level. I gave it a bunch of tools and it can search the internet, make product recommendations, document, code, etc.

Had the same reaction so had Grok create a script to: - find a free GPU droplet on digital ocean - fire it up - pull in a snapshot of the model + extra files/packages etc - set up a ssh tunnel so that the localhost:8000 routes to the above Then I just configured OpenCode to use the above and was off to the races. Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could…

Awesome, I used Claude to write a small python script to do the same with Linode's API. The only difference is I setup a persistent drive, and with Linode you can boot off of it. So my biggest start up lag is ~ 2 minutes to deploy + boot, then maybe 2 more to warm the model.

I actually dislike LLMs. But I'm a realist, and on-demand compute like this is massive cost saving measure.

(persistent drives are relatively cheap, compared to a box with several GPUs.. or even one. I find it worth the expense)

Re: Why your local LLM feels dumber than it is

#27

Post by embedding-shape »

JacobJack wrote: post #17

> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts. Genuine question : is there something fundamentally wrong with Ollama ? I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly). I thought the main advantage of VLLM was better concurrency management (better batching). But if the quality of the inte…

People who use Ollama generally (not everyone obviously) don't always clearly understand what quantization they use when running models, so people end up saying "I tried running Qwen 3.8 27b locally and it was dumb" while Ollama would default to a Q4 version of the model, which has very different results from the BF16 weights, doesn't really speak to the model itself because it's been so quantized in that case.

Sure, makes things easier, but tons of people misunderstand what they're using, then base and share their experiences on that, without really specifying what exact weights they use too.

For a single local user, using llama.cpp directly shouldn't be a problem if you're already using Ollama's CLI, it works basically the same except you manage weights yourself, and if you put your favorite agent to make sense of the faux "registry + image layers" Ollama has prepared locally for you, you can reuse the files you've already downloaded with Ollama.

Re: Why your local LLM feels dumber than it is

#28

Post by seanmcdirmid »

jonplackett wrote: post #13
prettyblocks wrote: post #4

Earlier quoted context omitted.

My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?

It’s hot and also LOUD and runs the battery down quick. But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in. It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever…

I have an m3 max 64GB and it runs ok, I was playing around with it last night. Its reasoning is better, but the token/s still makes an MoE model (35B A3B) preferable so I switched back.

Try using Goose instead of Claude's harness? Goose doesn't load as much context in at the start, so it might be more usable. I've definitely been able to get it past 128K, although I typically don't go beyond 70-80k. MoE might also be better at this.

Re: Why your local LLM feels dumber than it is

#29

Post by embedding-shape »

alexpotato wrote: post #16
Gareth321 wrote: post #12

Earlier quoted context omitted.

I tried it on my M1 MacBook Pro. It's slow but surprisingly smart as a general purpose LLM. Maybe GPT-5.3 level. I gave it a bunch of tools and it can search the internet, make product recommendations, document, code, etc.

Had the same reaction so had Grok create a script to: - find a free GPU droplet on digital ocean - fire it up - pull in a snapshot of the model + extra files/packages etc - set up a ssh tunnel so that the localhost:8000 routes to the above Then I just configured OpenCode to use the above and was off to the races. Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could…

> Works out to be about ~$2/hr all said

What GPU you end up with for that price? Vast.ai (https://cloud.vast.ai/?priceInstanceHourlyMax=2) has a bunch of setups available to reach 192GB VRAM under $2 :) Quick skim showed 4x48, 2x96 and 8x24, all for under 2 buckaroos or around there.

Re: Why your local LLM feels dumber than it is

#30

Post by cube00 »

JacobJack wrote: post #17

> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts. Genuine question : is there something fundamentally wrong with Ollama ? I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly). I thought the main advantage of VLLM was better concurrency management (better batching). But if the quality of the inte…

> is there something fundamentally wrong with Ollama ?

Yes https://news.ycombinator.com/item?id=47788385

Post reply on HN