Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why your local LLM feels dumber than it is

forum.level1techs.com

Post reply on HN
Print view View original

31–40 of 233 posts

Re: Why your local LLM feels dumber than it is

#31

Post by zepearl »

JacobJack wrote: post #17

> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts. Genuine question : is there something fundamentally wrong with Ollama ? I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly). I thought the main advantage of VLLM was better concurrency management (better batching). But if the quality of the inte…

I know nothing about "vLLM" -> anything better compared to "llama.cpp"?

I started with "Ollama" (precompiled version) and it worked and was good enough to understand the very basics.

Then I downloaded the sourcecode of "llama.cpp", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck with "llama.cpp" (and "ik_llama.cpp" in very few cases).

I honestly don't know what made "Ollama" (precompiled) so much slower than "llama.cpp" (compiled locally) at that time and I'm too lazy to doublecheck now, in any case I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload, which is the reason why I recommend it.

Re: Why your local LLM feels dumber than it is

#32

Post by smcleod »

b112 wrote: post #21
kangalioo wrote: post #19

Earlier quoted context omitted.

From what I've heard, Ollama has a bad reputation because it's a thin wrapper around llama.cpp without attributing it properly, thereby stealing recognition from the maintainers doing most of the work

It seems , and that seems is entirely my unvalidated impression, that Ollama lags in features, as they're integrating after the fact those changes. But (seriously) an LLM told me that, when some aspects of MoE models were better supported with the latest llama. And it did in that case make a significant difference.

It's very far behind llama.cpp, vLLM and SGLang in features yes. In part because of that but also due to some poor default settings it generally performs a lot worse as well.

Re: Why your local LLM feels dumber than it is

#33

Post by a1o »

catlifeonmars wrote: post #25

> I will make you read the really long unpleasant version with math. This is the version I want to read :) I assume it is unpleasant in spite of the math, not because of it?

I thought it was a link too because of the line under the with math but it isn’t. :/

Re: Why your local LLM feels dumber than it is

#34

Post by InvertedRhodium »

I’m running Qwen3.8 aggressive uncensored Q4_K_P on a 4090 in a loop against the 2026 CrackMe CTF challenges.

Using oh-my-pi in a prebuilt environment that I let Qwen build too.

Codex wouldn’t even look at the files - literally, as soon as it read something with CTF it shut down. Didn’t even offer to fall back to a dumber model.

Re: Why your local LLM feels dumber than it is

#35

Post by petcat »

dominotw wrote: post #18
alexchantavy wrote: post #9

Earlier quoted context omitted.

How many tok/s are you getting? What gen mbp?

i suspect ppl dropping generic "its awesome" comments are not actually using it and prbly just managed to get it running for a prompt or two.

Yeah, that's my experience. It's a big "wow" factor to get a non-trivial LLM running on my Mac, but it's actually not that useful. Like trying to use Photoshop at 8 FPS.

Re: Why your local LLM feels dumber than it is

#36

Post by VCFundedGenYer »

JacobJack wrote: post #17

> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts. Genuine question : is there something fundamentally wrong with Ollama ? I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly). I thought the main advantage of VLLM was better concurrency management (better batching). But if the quality of the inte…

In my limited testing of ollama (in a Docker container because I don't want Meta touching bare metal disk), I noticed that first of all the model is abandoned. Last updated in 2023. Secondly, it basically can't do anything. Anything I asked of it resulted in a lot of "I can't do that" so it's just stupid.

Re: Why your local LLM feels dumber than it is

#37

Post by ls612 »

prettyblocks wrote: post #4
jonplackett wrote: post #2

I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.

My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?

I tested Gemma 4 a couple months ago and yes it ramps the fans to max. Definitely a thing that should only be done on wall power.

Re: Why your local LLM feels dumber than it is

#38

Post by Macha »

VCFundedGenYer wrote: post #36
JacobJack wrote: post #17

> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts. Genuine question : is there something fundamentally wrong with Ollama ? I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly). I thought the main advantage of VLLM was better concurrency management (better batching). But if the quality of the inte…

In my limited testing of ollama (in a Docker container because I don't want Meta touching bare metal disk), I noticed that first of all the model is abandoned. Last updated in 2023. Secondly, it basically can't do anything. Anything I asked of it resulted in a lot of "I can't do that" so it's just stupid.

Are you confusing llama (the model from meta that popularised a lot of open AI stuff but is quite dated now) and ollama (the set of tools for running models, originally llama but no longer limited to it, not maintained by meta)

Re: Why your local LLM feels dumber than it is

#39

Post by CamperBob2 »

InvertedRhodium wrote: post #34

I’m running Qwen3.8 aggressive uncensored Q4_K_P on a 4090 in a loop against the 2026 CrackMe CTF challenges. Using oh-my-pi in a prebuilt environment that I let Qwen build too. Codex wouldn’t even look at the files - literally, as soon as it read something with CTF it shut down. Didn’t even offer to fall back to a dumber model.

How's it performing on the challenges?

Re: Why your local LLM feels dumber than it is

#40

Post by IronWolve »

sglang, 150+ tok/s on a 5090 in ubuntu 26.04 via wsl. gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090, dspark, medium reasoning, 96k context.

Using opencode and it built a old fashioned arcade vertical shooter with no issues.

Images are ok'ish, just had grok create updated images, and it came out great.

Post reply on HN