Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A guide to local coding models

aiforswes.com

Post reply on HN
Print view View original

201–210 of 363 posts

Re: A guide to local coding models

#201

Post by rubymamis »

thehamkercat wrote: post #24
simonw wrote: post #5

This story talks about MLX and Ollama but doesn't mention LM Studio - https://lmstudio.ai/ LM Studio can run both MLX and GGUF models but does so from an Ollama style (but more full-featured) macOS GUI. They also have a very actively maintained model catalog at https://lmstudio.ai/models

I think you should mention that LM Studio isn't open source. I mean, what's the point of using local models if you can't trust the app itself?

You can always use something like Little Snitch to not allow it to dial home.

Re: A guide to local coding models

#202

Post by [deleted] »

throwthrowuknow wrote: post #118
mudkipdev wrote: post #66

Earlier quoted context omitted.

Claude's $20 plan should be renamed to "trial". Try Opus and you will reach your limit in 10 minutes. With Sonnet, if you aren't clearing the context very often, you'll hit it within a few hours. I'm sympathetic to developers who are using this as their only AI subscription because while I was working on a challenging bug yesterday I reached the limit before it had even diagnosed the problem and had to switch to anot…

I half agree, but it should be called “Hobbiest” since that’s what it’s good for. 10 minutes is hyperbolic, I average 1h30m even when using plan mode first and front loading the context with dev diaries, git history, milestone documents and important excerpts from previous conversations. Something tells me your modules might be too big and need refactoring. That said, it’s a pain having to wait hours between sessions…

[deleted]

Re: A guide to local coding models

#203

Post by jwr »

I am still hoping, but for the moment… I have been trying every 30-80B model that came out in the last several months, with crush and opencode, and it's just useless. They do produce some output, but it's nowhere near the level that claude code gets me out of the box. It's not even the same league.

With LLMs, I feel like price isn't the main factor: my time is valuable, and a tool that doesn't improve the way I work is just a toy.

That said, I do have hope, as the small models are getting better.

Re: A guide to local coding models

#204

Post by BoredPositron »

SamDc73 wrote: post #145
eblanshey wrote: post #82

Earlier quoted context omitted.

What hardware can you buy for $5k to be able to run K2? That's a huge model.

This older HN thread shows R1 running on a ~$2k box using ~512 GB of system RAM, no GPU, at ~3.5-4.25 TPS: https://news.ycombinator.com/item?id=42897205 If you scale that setup and add a couple of used RTX 3090s with heavy memory offloading, you can technically run something in the K2 class.

Stop recommending 3090s they are all but obsolete now. Not having native bf16 is a showstopper.

Re: A guide to local coding models

#206

Post by avhception »

I tried local models for general-purpose LLM tasks on my Radeon 7800 XT (20GB VRAM), and was disappointed.

But I keep thinking: It should be possible to run some kind of supercharged tab completion on there, no? I'm spending most of my time writing Ansible or in the shell, and I have a feeling that even a small local model should give me vastly more useful completion options...

Re: A guide to local coding models

#207

Post by Eupolemos »

davidw wrote: post #54
slicktux wrote: post #51

Earlier quoted context omitted.

Agreed, Everyone seems to shun the DIY hacker now a days; saying things like “I’ll just pay for it”. It’s not about just NOT paying for it but doing it yourself and learning how to do it so that you can pass the knowledge on and someone else can do it.

I loathe the idea of being beholden to large corporations for what may be a key part of this job in the future.

And we all know that enshittyfication is coming.

Re: A guide to local coding models

#208

Post by cloudhead »

dkdcio wrote: post #100
cloudhead wrote: post #49

Earlier quoted context omitted.

For sure, and I guess that's kind of my point -- if the OP says local coding models are now good enough, then it's probably because he's using things that are towards the middle of the distribution.

similar for me —- also how do you get the proper double dashes —- anyway, I’d love to be able to run CLI agents fully local, but I don’t see it being good enough (relative to what you can get for pretty cheap from SOTA models) anytime soon

What’s wrong with your keyboard haha
Post reply on HN