Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why your local LLM feels dumber than it is

forum.level1techs.com

Post reply on HN
Print view View original

111–120 of 233 posts

Re: Why your local LLM feels dumber than it is

#111

Post by b112 »

aktenlage wrote: post #106
tharkun__ wrote: post #72

Earlier quoted context omitted.

It was actually great. I have like a non-AI box so to speak 8GB VRAM, co-incidentally from a gaming PC ... All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period. Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev` . while an…

Have you tried a mixture of experts model? Dense models have been quite slow for me, as I have only 6 GB VRAM. But with llama.cpp and --cpu-moe I get 200 t/s input and almost 30 t/s output with Gemma 4 26B A3B, which feels ok to use. Would be interested about your mileage there.

I wish qwen3.8 had a MoE variant, but the skinny is it won't be coming.

Re: Why your local LLM feels dumber than it is

#113

Post by b112 »

LeifCarrotson wrote: post #69
coldtea wrote: post #61

Earlier quoted context omitted.

> It’s hot and also LOUD and runs the battery down quick. Who'd run this on battery? Or do you mean kills the battery faster even when used while charging?

There was a lovely window of a few years when processors were fast enough and low-power enough that real development work could trivially happen on a Macbook Air in a lounge. That window is waning as more and more memory and graphics processing power is being used locally. I suppose the future is most likely going to involve farming out AI requests to your desktop machine, your company's compute farm, or a cloud endp…

I envision a future when the potential energy in a laptop battery, could take out 17 city blocks if it exploes.

Re: Why your local LLM feels dumber than it is

#114

Post by serbuvlad »

happybox2016 wrote: post #109

Rate limiting on free LLM APIs is usually where the pain lies. I've seen 5 concurrent reqs hit 20K/day limit in under 2 hours. Does anyone know a free API that still allows some reasonable concurrent requests?

Asking for free inference is like asking for free gold in today's economy. :)

Re: Why your local LLM feels dumber than it is

#115

Post by heywoods »

So to what extent does this apply to cloud hosted LLM’s? Are there benchmarks that score models across cloud providers? My experience using LLM’s during day time vs evening sessions has felt “night and day” and I’ve chalked that mostly up to it must be my imagination or just the general indeterministic nature of LLM’s. Sessions resumed after a day away also feel “dumb” sometimes so I can see an aggressive kv cache eviction policy playing a role if it’s reasonable to extrapolate what the article is saying about local inference.

Are things like kv cache eviction policies and memory budgets shipped with recommended configurations based on the hardware and software serving the inference requests? or are they configured dynamically by the cloud provider hosting the model to manage multi-tenant load?

Re: Why your local LLM feels dumber than it is

#116

Post by funlang »

raffraffraff wrote: post #110
stymaar wrote: post #105

Earlier quoted context omitted.

> It took him two hours of passing errors to Claude for the endpoint to start working What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with. I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).

I'm not saying this is the case in the particular example, but these days a lot of people aren't bothering to read a 1 page README, and are instead letting claude "just fucking do it already". I see it at work. Claude has got better at "just fucking doing it" by asking if it's ok to go read the latest github issues and pull the README, which means that people will likely get lazier and lazier.

[flagged]

Re: Why your local LLM feels dumber than it is

#117

Post by alibarber »

pmontra wrote: post #91
dmitrygr wrote: post #82

Earlier quoted context omitted.

No idea what you are talking about. My battery lasts longer than ever while running vim and make and GCC. It’s amazing. Not sure why your windows are closing.

Because the local LLM, which you are not running, is running for much longer than gcc and is eating the battery. Different choices, different outcomes.

Bur the implication here in the parent comment is that for ‘real dev work’ to be done you must be running an LLM, and therefore you can’t do it on a MacBook anymore.

Turns out you can still accomplish stuff with a text editor and compiler.

Re: Why your local LLM feels dumber than it is

#118

Post by WhyNotHugo »

fxtentacle wrote: post #108
stymaar wrote: post #105

Earlier quoted context omitted.

> It took him two hours of passing errors to Claude for the endpoint to start working What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with. I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).

My experience with Claude is that it suffers badly from “not invented here” syndrome. So probably it rebuilt something like llama from scratch and then 2 hours suddenly seems reasonable (if you don’t question the approach). And that’s the thing, someone with no experience isn’t going to question it.

Not necessarily rebuild llama from scratch, build attempting to build it without cmake and manually invoking all the build commands would be quite in character.

Re: Why your local LLM feels dumber than it is

#119

Post by rasitakyol »

JacobJack wrote: post #17

> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts. Genuine question : is there something fundamentally wrong with Ollama ? I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly). I thought the main advantage of VLLM was better concurrency management (better batching). But if the quality of the inte…

[flagged]

Re: Why your local LLM feels dumber than it is

#120

Post by noman-land »

velcrovan wrote: post #80
jonplackett wrote: post #2

I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.

That's funny, I downloaded the same model on my 48GB M4 Pro and gave it a problem to solve in an existing codebase, it spun its wheels for twenty minutes and then fell over dead. This was using LMStudio and pi as a harness; I never use pi for anything else, so maybe I'm holding it wrong.

I've recently learned and then observed that oMLX serves local models much, much faster than LM Studio.
Post reply on HN