Live data from Hacker News

AI coding at home without going broke

stephen.bochinski.dev

121–130 of 321 posts

Re: AI coding at home without going broke

#121
post #28

I think someone could find some way to use the smaller local models to write code. Some kind of framework or harness or language or something. But not too many people are working on that because the big models are pretty cheap and a lot better.

Maybe one possible path(to make weaker models highly capable) is making the job of the llm as easy as possible. I wonder if part of the solution is building/finding the right libraries, with the right documentation/language/API(one that plays well with LLM's) and maybe creating some synthetic data around them - to make it very easy for the llm. And maybe there could be a business model around creating those libraries…

So in my limited experience: The smaller the model, the bigger the harness. The biggest issue becomes the context window. For big models you can kind of just give it bash access and let it run... while with the smaller ones you need to fully manage the context in each LLM call.

If you can ask the model for a specific function; with a spec design (typed languages help too) then the small models are great! I have had good progress with generating small python modules for example, but you need verification rounds to catch issues.

So test driven design + a good spec sheet + a very detailed todo.md (or even better if its todo.json because then the LLM does not need to manage it, you do from the harness) is your best bet for small models.

Re: AI coding at home without going broke

#122
post #98

What are people doing at home? I have like 5 different apps I code on the $20/month Claude plan and like sure I can hit rate limits but - What are people doing to burn through $3k in tokens?

YMMV but automations eat through the $100-$200 plans, which burn thousands in tokens alone.

I have hourly automations for root cause analysis on customer support issues, daily automations for eg log analysis, weekly & monthly automations for KPI tracking & actioning.

I will say, when I was building side projects that were 1) fairly well defined in scope and 2) without users/need for automations it was much easier to stay under $20/mo plan limits. Now I regularly hit weekly limits and need multiple Max plans

Re: AI coding at home without going broke

#123
post #76

I invested about $4,000 in an NVIDIA DGX Spark several months ago. 128 GB of unified RAM, and the NVIDIA GB10 chip. With the RAM, the several CPU cores, and the 4 TB NVMe SSD, it's a very capable ARM64 Linux computer even without the GPU, and so far I've mostly been using it as such. But I wonder, what's the most capable model, specifically for coding, that can run well on that hardware?

https://www.canirun.ai/?status=tight might answer that question

That site doesn't seem to consider the quants. So useless.

Re: AI coding at home without going broke

#125
post #120

>Around $400 a month of plans buys roughly $2800 of API usage at list prices, which is a real bargain right up until you hit the ceiling. The plans are metered, and any large AI native workflow will chew through the included tokens fast I don't think that's true at all. I'm doing 8-12 PRs a week at work, all primarily Claude Code, and the usage at API billing has never broken $500/mo.

Maybe yours is not a large AI native workflow?

[deleted]

Re: AI coding at home without going broke

#126
post #48
post #3

> The first is to self host. You buy the machine, run open source models locally, and pay nothing per token after that. Power is not free. What I’ve found is that you’re basically paying a premium for privacy, and that’s worth it for me.

Luckily I needed a new laptop and I bought an M1 Max secondhand from a friend quite cheaply because it was fast enough to recompile something else I am interested in. So for me, there is no additional hardware cost; it was acquired in replacement. I run the AI models at home on this kit because I want to; I'll use openrouter if I need to. I accept the economics of this article are right. But I feel so incredibly sad…

You sound awesome. Just venting? (b/c curious if friends can fill your heart abundantly, & we know we're never too old to make new friends!)

> dreading

Even avoiding political headlines (OK, at least articles), plenty of cause for dread, so I keep re-focusing to avoid despair. Easier said than done innit!

Can't kill my hope for the future though. One day, all the good stuff shall prevail (morality, intelligence, love & kindness)... maybe not permanently, but a Star Trek future is there somewhere (& they had their troubles but it wouldn't be a dreadful situation overall). Sharing with you in case it's even slightly contagious!

Re: AI coding at home without going broke

#127

I feel like I must have plateued and don't know what to do next to level up. I'm currently on the $100/month codex plan and it seems fine using 5.5-xhigh all the time. I think of what to do next, have a chat session to determine exactly what to ask for up to the point of being ready to implement, and then codex churns on a commit-sized task whereupon I briefly check it on my local dev server. If necessary I ask for a…

I’ve watched a bunch of layman videos where they create stuff with AI, these people burning through 12 hour tasks are literally not reading the output or understanding what it’s doing. Like they’ll ask for a program, and then right after it’s been created they ask the AI how to run it. Then when there’s a bug, they ask the AI what went wrong, or scrap the entire thing and switch model/harness and try again. Here’s an…

Amazingly, there are people out there (apart from creators), that work that way in their day-to-day job. I had the pleasure to work with such a person. After several months, he got removed from the position. He left a mess that hasn't been cleaned up completely to this point.

Re: AI coding at home without going broke

#128
post #86

Earlier quoted context omitted.

Yeah, although that is pushing every rate limit and no one knows what happens if you do that consistently? I think $4,000/mo is probably a good estimate for an individual dev doing synchronous coding agent work.

Yeah, I agree. I've been consistently getting about $1,000/month of value out of the $100/month subscription for OpenAI, and about the same for Anthropic.

Sorry to be that guy. I think the more precise wording would be that you get tokens which would cost $1,000/month at API pricing. Maybe (depending on the profit margin of the API pricing) you incur costs somewhere close to $1,000/month. And maybe your usage is subsidized by 900$/month. The value you get out of it is a whole other question. One that according to recent news, CFOs find hard to esitimate.

Re: AI coding at home without going broke

#129

Earlier quoted context omitted.

I’ve watched a bunch of layman videos where they create stuff with AI, these people burning through 12 hour tasks are literally not reading the output or understanding what it’s doing. Like they’ll ask for a program, and then right after it’s been created they ask the AI how to run it. Then when there’s a bug, they ask the AI what went wrong, or scrap the entire thing and switch model/harness and try again. Here’s an…

Amazingly, there are people out there (apart from creators), that work that way in their day-to-day job. I had the pleasure to work with such a person. After several months, he got removed from the position. He left a mess that hasn't been cleaned up completely to this point.

It won’t be long till employers get wise to this stuff, they just need to burned a couple of times.

It seems AI is good, great even at many things. But it doesn’t seem like it’s going to change the world as much as some people believe it will. And if it does it’s going to take time

Re: AI coding at home without going broke

#130

Earlier quoted context omitted.

I’ve watched a bunch of layman videos where they create stuff with AI, these people burning through 12 hour tasks are literally not reading the output or understanding what it’s doing. Like they’ll ask for a program, and then right after it’s been created they ask the AI how to run it. Then when there’s a bug, they ask the AI what went wrong, or scrap the entire thing and switch model/harness and try again. Here’s an…

Amazingly, there are people out there (apart from creators), that work that way in their day-to-day job. I had the pleasure to work with such a person. After several months, he got removed from the position. He left a mess that hasn't been cleaned up completely to this point.

Yeesh that sounds painful. There's definitely a fine line between vibe coding as a professional engineer and vibe coding as an outsider.
Post reply on HN