[flagged]
Why your local LLM feels dumber than it is
171–180 of 233 posts
Re: Why your local LLM feels dumber than it is
#172Earlier quoted context omitted.
Had the same reaction so had Grok create a script to: - find a free GPU droplet on digital ocean - fire it up - pull in a snapshot of the model + extra files/packages etc - set up a ssh tunnel so that the localhost:8000 routes to the above Then I just configured OpenCode to use the above and was off to the races. Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could…
> Works out to be about ~$2/hr all said What GPU you end up with for that price? Vast.ai ( https://cloud.vast.ai/?priceInstanceHourlyMax=2 ) has a bunch of setups available to reach 192GB VRAM under $2 :) Quick skim showed 4x48, 2x96 and 8x24, all for under 2 buckaroos or around there.
It's actually a bit less than $2 but I rounded up just to make the math easier.
And yeah, did a lot of work with Vast AI at a past job and it's pretty wild the variety of prices/hardware that they have.
Re: Why your local LLM feels dumber than it is
#173Earlier quoted context omitted.
Had the same reaction so had Grok create a script to: - find a free GPU droplet on digital ocean - fire it up - pull in a snapshot of the model + extra files/packages etc - set up a ssh tunnel so that the localhost:8000 routes to the above Then I just configured OpenCode to use the above and was off to the races. Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could…
Awesome, I used Claude to write a small python script to do the same with Linode's API. The only difference is I setup a persistent drive, and with Linode you can boot off of it. So my biggest start up lag is ~ 2 minutes to deploy + boot, then maybe 2 more to warm the model. I actually dislike LLMs. But I'm a realist, and on-demand compute like this is massive cost saving measure. (persistent drives are relatively ch…
> the only difference is I setup a persistent drive,
Going to try this out vs the snapshot!
I also really like this experiment b/c it's a mix of LLMs and old school IaaC/DevOps.
Re: Why your local LLM feels dumber than it is
#174Earlier quoted context omitted.
Add Just [...] And you have also fully adopted the writing patterns.
Seems like he's self-aware, though - he's taking steps to move away from brain-atrophy.
My ability to detect sarcasm is not good. From looking at funlang's profile and other comments the profile looks like a LLM generated bot.
Forums with full no verification pseudonyms seem like they have a real challenge ahead. How long until we need humanhackernews.com with public pseudonyms and a private trusted verification?
Re: Why your local LLM feels dumber than it is
#175I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.
Which exact model are you running? With only 48GB of RAM, by the time I got a model small enough, it was pretty bad in performance both in speed and reasoning.
Re: Why your local LLM feels dumber than it is
#176Earlier quoted context omitted.
Because the local LLM, which you are not running, is running for much longer than gcc and is eating the battery. Different choices, different outcomes.
Bur the implication here in the parent comment is that for ‘real dev work’ to be done you must be running an LLM, and therefore you can’t do it on a MacBook anymore. Turns out you can still accomplish stuff with a text editor and compiler.
Of course we can code without a LLM, but if we want to, as the sibling comment says, don't run a local LLM on a laptop. I'm offloading the LLM to the LLM vendor, like most of us IMO. The CPU is still busier then it used to be, because of the local harness.
Re: Why your local LLM feels dumber than it is
#177Earlier quoted context omitted.
It was actually great. I have like a non-AI box so to speak 8GB VRAM, co-incidentally from a gaming PC ... All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period. Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev` . while an…
I'm having the same constraint, 8GB VRAM, and was trying the past months to get a local model running to drive my Home Assistant setup via voice, so many tool calls, little room for errors. I've tried multiple models, different temperature and top p values, different context windows... turns out that a quantized gemma4 is the best pick so far but qwen3.5:9b isn't far off. Surprisingly newer models are not necessarily…
I've had great success using Gemma4 even on old hardware (4GB VRAM). I haven't found a Qwen model that can match it on very low end hardware.
Re: Why your local LLM feels dumber than it is
#178Any DGX Spark users in this thread? What's your favourite model to run on it?
Without doubt, dsv4-flash-0731. Original weights; needs two connected DGX.
Re: Why your local LLM feels dumber than it is
#179Earlier quoted context omitted.
> Works out to be about ~$2/hr all said What GPU you end up with for that price? Vast.ai ( https://cloud.vast.ai/?priceInstanceHourlyMax=2 ) has a bunch of setups available to reach 192GB VRAM under $2 :) Quick skim showed 4x48, 2x96 and 8x24, all for under 2 buckaroos or around there.
It's actually a bit less than $2 but I rounded up just to make the math easier. And yeah, did a lot of work with Vast AI at a past job and it's pretty wild the variety of prices/hardware that they have.
> It's actually a bit less than $2 but I rounded up just to make the math easier.
That's cool, what actual GPU though? I'm still curious :P
Re: Why your local LLM feels dumber than it is
#180There's quite a few tangential features that must be implemented correctly or risk affecting the LLM output in significant ways. Parsing/encoding is one example: A couple of months ago I've debugged a reasoning loop bug in Step 3.7 Flash on llama.cpp that was caused by the parser capturing an extra `\n` as part of a reasoning block. It was something that only manifested at longer multi-turn agentic sessions, and the…
This is fascinating. I’m struggling to understand how that was causing such a large difference in the output. Is the “autoparser” vulnerable to injections somehow? How do you distinguish between user text, model text, and metadata, or is there ambiguity in the parsing?