401–410 of 620 posts
I just use DeepSeekV4 Fast... It's cheap as hell. Currently my monthly usage has been 67M Ouput 51M Input Total $0.83 dollar. I honestly don't understand why people just don't use DeepSeek.
You mean deepseek-v4-flash, right? Same here. I use it for my Hermes agent. It's so cheap that I sometimes feel "guilty". I even put more money than I needed just make sure they do not go out of business.
But, guys, when you say Claude/ GPT models, do you stop to think what are these "models"? One day I thought about how can GPT send thinking parts one after another with a markdown header summary of the thinking block itself. Just think about it. As a matter of fact, think about these operations, api endpoints, observe their output. These so called SOTA models are not what meets the eye, and are not at all comparable…
Earlier quoted context omitted.
Hopefully this isn't off-topic, but your setup sounds just like mine, Strix Halo and (I'm assuming) llama.cpp on ROCm, and I'm finding that the Qwen hybrid models don't handle prompt caching and instead re-process the context in full on every turn. I'm wondering if you were able to solve this and how?
I use Vulkan mostly instead of ROCm. Vulkan is actually a bit faster, paradoxically. I do switch out and try them both out, and it's not a huge difference, but I've been mostly saying on Vulkan. The re-processing context every turn problem is definitely something I've hit. Some of the causes have been solved upstream in llama.cpp; make sure you're up to date. But another cause of the issue that has a big effect is th…
but for caching, all you are doing is leaving off a fraction of the most recent assistant message generation, which will have little/no impact on cache hit rate.
Until I can buy an 80GB VRAM GPU, I won't attempt to do it. A local LLM is always missing something that needs a bigger model.
Which model class requires an 80 GB VRAM GPU? From my perspective, popular models seem to be either in the ~30B range (Qwen3.6, Gemma 4), while the larger models (MiniMax, MiMo, StepFun, Deepseek) are in the multiple hundreds of billions parameters, for which 80 GB is simply too small. You can just about reach the lower end of the latter category with a 128GB machine like a DGX Spark, Framework Desktop, or M5 Max, th…
Earlier quoted context omitted.
Hopefully this isn't off-topic, but your setup sounds just like mine, Strix Halo and (I'm assuming) llama.cpp on ROCm, and I'm finding that the Qwen hybrid models don't handle prompt caching and instead re-process the context in full on every turn. I'm wondering if you were able to solve this and how?
There is a bug in llama-cpp for qwen/gemma models, use vLLM instead
Earlier quoted context omitted.
But you need the best tools to do the job
You need tools sufficient to do the job in an economical way, optimizing for both cost and quality. That is what 'best' means. We don't give every engineer all the resources under the sun, only what is appropriate. I suspect many will realize millions more dollars are being spent than needed to achieve the highest marginal productivity gains, and reallocate accordingly. Who wants more of their money going to develope…
That's way more economical and produces far better result than any self hosted models today.
But, guys, when you say Claude/ GPT models, do you stop to think what are these "models"? One day I thought about how can GPT send thinking parts one after another with a markdown header summary of the thinking block itself. Just think about it. As a matter of fact, think about these operations, api endpoints, observe their output. These so called SOTA models are not what meets the eye, and are not at all comparable…
This all sounds very mysterious
Yes. Llama.cpp + Qwen3.6-35b (MTP) + OpenCode is quite capable and runs on a single RTX 3090 and is faster than most cloud models. Quality is like running edge models from 8-12 months ago. Setup details at https://github.com/pierotofy/LocalCodingLLM/
"Quality is like running edge models from 8-12 months ago." That sounds great for hobbyists but IMHO it wasn't until Opus 4.6 was released six months go (Dec 25, 2025) that we had a model good enough for professionals to use as a primary driver of their coding agents. That seems to be the threshold worth aiming for.
Certainly I get a ton more value out of Opus today, but I could absolutely see someone deciding to limit themselves to 8-to-12-months-ago Opus performance for privacy (or other) reasons.
Earlier quoted context omitted.
> Comparing agentic Qwen3.6 35b to Claude Opus is like a junior with knowledge across the board, that you really need to guide, versus a senior that thinks with you on architecture. that's why i use the frontier models because its a senior co-worker vs a junior. if you use the junior for the sake of privacy i think you're missing out on the best insights for a specific task.
The dilemma I am facing is cost. Consumer-grade subscriptions of the frontier models give you superb capabilities per dollar, them being heavily subsidized. But if you're working in an enterprise setting, that won't work. You need to upgrade, and that gets significantly more expensive. Furthermore, basing the SDLC on leveraging the bargain subscriptions risks falling apart in the future, both from a cost perspective…