Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A guide to local coding models

aiforswes.com

Post reply on HN
Print view View original

171–180 of 363 posts

Re: A guide to local coding models

#171

Post by r-w »

kristopolous wrote: post #166
m4ck_ wrote: post #108

Earlier quoted context omitted.

Is your RAG manpages thing on github somewhere? I was thinking about doing something like that (it's high on my to-do list but I haven't actually done anything with llms yet.)

I'll get it up soon, probably should. This little snippet will help you though: $ man --html="$(which markitdown)" That goes man -> html -> markdown which is not only token efficient but also llms are pretty good at creating hierarchies from markdown

I bet you could do the same thing with pandoc and skip serializing to HTML entirely.

Re: A guide to local coding models

#172

Post by 1over137 »

Aurornis wrote: post #159
selcuka wrote: post #141

Earlier quoted context omitted.

Not the same poster, but apparently they tried the $200/mo subscription, but after seeing they don't need it, they "subscribed to all three at their lowest subscriptions (for $60/mo)" instead.

> but apparently they tried the $200/mo subscription, but after seeing they don't need it This is why it’s confusing, though. Why start with the highest plan as the starting point when it’s so easy to upgrade?

Because you’re rich?

Re: A guide to local coding models

#173

Post by nl »

amarant wrote: post #152

Buying a maxed out MacBook Pro seems like the most expensive way to go about getting the necessary compute. Apple is notorious for overcharging for hardware, especially on ram. I bet you could build a stationary tower for half the price with comparable hardware specs. And unless I'm missing something you should be able to run these things on Linux. Getting a maxed out non-apple laptop will also be cheaper for compara…

You want unified RAM.

On Linux your options are the NVidia Spark (and other vendor versions) or the AMD Ryzen AI series.

These are good options, but there are significant trade-offs. I don't think there are Ryzen AI laptops with 128GB RAM for example, and they are pricey compared to traditional PCs.

You also have limited upgradeability anyway - the RAM is soldered.

Re: A guide to local coding models

#174

Post by alok-g »

brainless wrote: post #149

I do not spend $100/month. I spend for 1 Claude Pro subscription and then a (much cheaper) z.ai Coding Plan, which is like one fifth the cost. I use Claude for all my planning, create task documents and hand over to GLM 4.6. It has been my workhorse as a bootstrapped founder (building nocodo, think Lovable for AI agents).

I have heard about this approach elsewhere too. Could you please provide some more details on the set up steps and usage approach. I would like to replicate. Thanks.

Re: A guide to local coding models

#175

Post by aquafox »

kristopolous wrote: post #91
simonw wrote: post #4

> I realized I looked at this more from the angle of a hobbiest paying for these coding tools. Someone doing little side projects—not someone in a production setting. I did this because I see a lot of people signing up for $100/mo or $200/mo coding subscriptions for personal projects when they likely don’t need to. Are people really doing that? If that's you, know that you can get a LONG way on the $20/month plans fr…

I use local models + openrouter free ones. My monthly spend on ai models is I'm not cheap, just ahead of the curve. With the collapse in inference cost, everything will be this eventually I'll basically do $ man tool | or even $ cat source | Things I used to do intensively I now do lazily. I've even made a IEITYuan/Yuan-embedding-2.0-en database of my manpages with chroma and then I can just ask my local documentatio…

> I'll basically do

    $ man tool |
or even $ cat source |

Could you please elaborate on this? Do I get this right that you can set up your your command line so that you can pipe something to a command that sends this something together with a question to an LLM? Or did you just mean that metaphorically? Sorry if this is a stupid question.

Re: A guide to local coding models

#176

Post by nl »

SamDc73 wrote: post #145
eblanshey wrote: post #82

Earlier quoted context omitted.

What hardware can you buy for $5k to be able to run K2? That's a huge model.

This older HN thread shows R1 running on a ~$2k box using ~512 GB of system RAM, no GPU, at ~3.5-4.25 TPS: https://news.ycombinator.com/item?id=42897205 If you scale that setup and add a couple of used RTX 3090s with heavy memory offloading, you can technically run something in the K2 class.

Is 4 TPS actually useful for anything?

That's around 350,000 tokens in a day. I don't track my Claude/Codex usage, but Kilocode with the free Grok model does and I'm using between 3.3M and 50M tokens in a day (plus additional usage in Claude + Codex + Mistral Vibe + Amp Coder)

I'm trying to imagine a use case where I'd want this. Maybe running some small coding task overnight? But it just doesn't seem very useful.

Re: A guide to local coding models

#178

Post by hu3 »

selcuka wrote: post #147
artursapek wrote: post #128

Imagine buying hardware that will be obsolete in 2 years instead of paying Anthropic $200 for $1000+ worth of tokens per month

> Imagine buying hardware that will be obsolete in 2 years Unless the PC you buy is more than $4,800 (24 x $200) it is still a good deal. For reference, a MacBook M4 Max with 128GB of unified RAM is $4,699. You need a computer for development anyway, so the extra you pay for inference is more like $2-3K. Besides, it will still run the same model(s) at the same speed after that period, or even maybe faster with future…

The value depreciation of the hardware alone is going to be significant. Probably enough to pay for 3x ~$20 subscriptions to OpenAI, Anthropic and Gemini.

Also, if you use the same mac to work, you can't reserve all 128GB for LLMs.

Not to mention a mac will never run SOTA models like Opus 4.5 or Gemini 3.0 which subscriptions gives you.

So unless you're ready to sacrifice quality and speed for privacy, it looks like a suboptimal arrangement to me.

Re: A guide to local coding models

#179

Post by ikidd »

So I can't see bothering with this when I pumped 260M tokens through running in Auto mode on a $20/mo Cursor plan. It was my first month of a paid subscription, if that means anything. Maybe someone can explain how this works for them?

Frankly, I don't understand it at all, and I'm waiting for the other shoe to drop.

Post reply on HN