Earlier quoted context omitted.
This is essentially vast.ai, no?
A quick glance at their homepage says they run in "secure datacenters", so no.
Then you glanced too quickly, vast.ai absolutely has non-datacenter GPUs.
A guide to local coding models
221–230 of 363 posts
Re: A guide to local coding models
#222Here's my take on it though...
Just as we had the golden era of the internet in the late 90s, when the WWW was an eden of certificate-less homepages with spinning skulls on geocities without ad tracking, we are now in the golden era of agentic coding where massive companies make eye watering losses so we can use models without any concerns.
But this won't last and Local Llamas will become a compelling idea to use, particularly when there will be a big second hand market of GPUs from liquidated companies.
Re: A guide to local coding models
#223I don’t think I’ve ever read an article where the reason I knew the author was completely wrong about all of their assumptions was that they admitted it themselves and left the bad assumptions in the article. The above paragraph is meant to be a compliment. But justifying it based on keeping his Mac for five years is crazy. At the rate things are moving, coding models are going to get so much better in a year, the ga…
Yes, it’s worth it, if only because that Mac will be worth $20k in 3 months…
Do you think prices will go up for mac?
Re: A guide to local coding models
#224Earlier quoted context omitted.
Is that really the case? This summer there was "Frontier AI performance becomes accessible on consumer hardware within a year" [1] which makes me think it's a mistake to discount the open weights models. [1] https://epoch.ai/data-insights/consumer-gpu-model-gap
Open weight models are neat. But for SOTA performance you need specialized hardware. Even for Open Weight models. 40k in consumer hardware is never going to compete with 40k of AI specialized GPUs/servers. Your link starts with: > "Using a single top-of-the-line gaming GPU like NVIDIA’s RTX 5090 (under $2500), anyone can locally run models matching the absolute frontier of LLM performance from just 6 to 12 months ago…
> I highly doubt a RTX 5090 can run anything that competes with Sonnet 3.5 which was released June, 2024.
I don't know about the capabilities of a 5090 but you probably can run a Devstral-2 [1] model locally on a Mac with good performance. Even the small Devstral-2 model (24b) seems to easily beat Sonnet 3.5 [2]. My impression is that local models have made huge progress.
Coding aside I'm also impressed by the Ministral models (3b, 8b and 14b) Mistral AI released a a couple of weeks ago. The Granite 4.0 models by IBM also seem capable in this context.
Re: A guide to local coding models
#225Earlier quoted context omitted.
> Imagine buying hardware that will be obsolete in 2 years Unless the PC you buy is more than $4,800 (24 x $200) it is still a good deal. For reference, a MacBook M4 Max with 128GB of unified RAM is $4,699. You need a computer for development anyway, so the extra you pay for inference is more like $2-3K. Besides, it will still run the same model(s) at the same speed after that period, or even maybe faster with future…
The value depreciation of the hardware alone is going to be significant. Probably enough to pay for 3x ~$20 subscriptions to OpenAI, Anthropic and Gemini. Also, if you use the same mac to work, you can't reserve all 128GB for LLMs. Not to mention a mac will never run SOTA models like Opus 4.5 or Gemini 3.0 which subscriptions gives you. So unless you're ready to sacrifice quality and speed for privacy, it looks like…
I suspect depreciation will be a bit slower for a while, because there is a supply crunch.
Re: A guide to local coding models
#226> I realized I looked at this more from the angle of a hobbiest paying for these coding tools. Someone doing little side projects—not someone in a production setting. I did this because I see a lot of people signing up for $100/mo or $200/mo coding subscriptions for personal projects when they likely don’t need to. Are people really doing that? If that's you, know that you can get a LONG way on the $20/month plans fr…
The limits for the $20/month plan can be reached in 10-20 minutes when having it explore large codebases with directed. It’s also easy to blow right through the quota if you’re not managing content well (waiting until it fills up and then auto-compacting, or even using /compact frequently instead of /clear or the equivalent in different tools). For most of my work I only need the LLM to perform a structured search of…
You'd think local models could explore a codename and build up a knowledge graph of it they could use to query it.
It could take longer, but save your subscription tokens.
Re: A guide to local coding models
#227Earlier quoted context omitted.
Claude's $20 plan should be renamed to "trial". Try Opus and you will reach your limit in 10 minutes. With Sonnet, if you aren't clearing the context very often, you'll hit it within a few hours. I'm sympathetic to developers who are using this as their only AI subscription because while I was working on a challenging bug yesterday I reached the limit before it had even diagnosed the problem and had to switch to anot…
> With Sonnet, if you aren't clearing the context very often, you'll hit it within a few hours. Do you mean that users should start a new chat for every new task, to save tokens? Thanks.
Yes, it also helps keep it focused.
Re: A guide to local coding models
#228I do not spend $100/month. I spend for 1 Claude Pro subscription and then a (much cheaper) z.ai Coding Plan, which is like one fifth the cost. I use Claude for all my planning, create task documents and hand over to GLM 4.6. It has been my workhorse as a bootstrapped founder (building nocodo, think Lovable for AI agents).
I have heard about this approach elsewhere too. Could you please provide some more details on the set up steps and usage approach. I would like to replicate. Thanks.
I simply ask Claude Sonnet, using claudecode, to use opencode. That's it! Example:
We need to clean up code lint and format errors across multiple files. Check which files are affected using cargo commands. Please use opencode, a coding agent that is installed. Use `opencode run ` to pass in a per-file prompt to opencode, wait for it to finish, check and ask again if needed, then move to next file. Do not work on files yourself.Re: A guide to local coding models
#229Earlier quoted context omitted.
LMStudio is not open source though, ollama is but people should use llama.cpp instead
I suspect Ollama is at least partly moving away open source as they look to raise capitol, when they released their replacement desktop app they did so as closed source. You're absolutely right that people should be using llama.cpp - not only is it truly open source but it's significantly faster, has better model support, many more features, better maintained and the development community is far more active.
Only issue I have found with llama.cpp is trying to get it working with my amd GPU. Ollama almost works out of the box, in docker and directly on my Linux box.
Re: A guide to local coding models
#230Earlier quoted context omitted.
On a $20/mo plan doing any sort of agentic coding you'll hit the 5hr window limits in less than 20 minutes.
With Codex it only happened to me once in my 4.5hr session here: https://simonwillison.net/2025/Dec/15/porting-justhtml/ Claude Code is a whole lot less generous though.
This is useful info.
I havent tried agentic coding as I havent set it up in a container yet, and not going to yolo my system (doing stuff via chat and a utility to copy and paste directories and files got me pretty far over the last year and a half).