Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A guide to local coding models

aiforswes.com

Post reply on HN
Print view View original

291–300 of 363 posts

Re: A guide to local coding models

#291

Post by jgalt212 »

bilater wrote: post #289

If you are using local models for coding you are midwiting this. Your code should be worth more than a subscription. The only legit use case for local models is privacy. I don't know why anyone would want to code with an intern level model when they can get a senior engineer level model for a couple of bucks more. It DOESN'T MATTER if you're writing a simple hello world function or building out a complex feature. Jus…

I will use a local coding model for our proprietary / trade secrets internal code when Google uses Claude for its internal code and Microsoft starts using Gemini for internal code.

The flip side of this coin is I'd be very excited if Jane Street or DE Shaw were running their trading models through Claude. Then I'd have access to billions of dollars of secrets.

Re: A guide to local coding models

#292

Post by artursapek »

hu3 wrote: post #178
selcuka wrote: post #147

Earlier quoted context omitted.

> Imagine buying hardware that will be obsolete in 2 years Unless the PC you buy is more than $4,800 (24 x $200) it is still a good deal. For reference, a MacBook M4 Max with 128GB of unified RAM is $4,699. You need a computer for development anyway, so the extra you pay for inference is more like $2-3K. Besides, it will still run the same model(s) at the same speed after that period, or even maybe faster with future…

The value depreciation of the hardware alone is going to be significant. Probably enough to pay for 3x ~$20 subscriptions to OpenAI, Anthropic and Gemini. Also, if you use the same mac to work, you can't reserve all 128GB for LLMs. Not to mention a mac will never run SOTA models like Opus 4.5 or Gemini 3.0 which subscriptions gives you. So unless you're ready to sacrifice quality and speed for privacy, it looks like…

Yeah, didn't even mention the fact that you can't Opus on your own hardware. Total waste of cash.

Re: A guide to local coding models

#293

Post by pcl »

bilater wrote: post #289

If you are using local models for coding you are midwiting this. Your code should be worth more than a subscription. The only legit use case for local models is privacy. I don't know why anyone would want to code with an intern level model when they can get a senior engineer level model for a couple of bucks more. It DOESN'T MATTER if you're writing a simple hello world function or building out a complex feature. Jus…

Or if you want to do development work while offline.

Re: A guide to local coding models

#294

Post by BoredPositron »

qayxc wrote: post #274
BoredPositron wrote: post #253

Earlier quoted context omitted.

If you run fp32 maybe but no sane person does that. The tensor performance of the 3090 is also abysmal. If you run bf16 or fp8 stay away from obsolete cards. Its barely usable for llms and borderline garbage tier on video and image gen.

Actual benchmarks show otherwise. > The tensor performance of the 3090 is also abysmal. I for one compared my 50-series card's performance to my 3090 and didn't see "abysmal performance" on the older card at all. In fact, in actual real-world use (quantised models only, no one runs big fp32 models locally), the difference in performance isn't very noticeable at all. But I'm sure you'll be able to provide actual numbe…

To be clear, we are not discussing small toy models but to be fair I also don't use consumer cards. Benchmarks are out there (phoronix, runpod, hugginface or from Nvidias own presentation) and they say it's at least 2x on high and nearly 4x on low precision, which is comparable to the uplift I see on my 6000 cards, if you don't see the performance uplift everyone else sees there is something wrong with your setup and I don't have the time to debug it.

Re: A guide to local coding models

#295

Post by cmrdporcupine »

fragmede wrote: post #234
beepbooptheory wrote: post #93

Earlier quoted context omitted.

Why would you ever hire someone to help with a personal open source project?

because we want to support open source? Even if you're independence maximalist, you still pay other people in your life to do things for you at some point. If you've got the money and the desire but not the time, why does that not seem reasonable to you?

Frankly I almost consider it a duty to use these agents -- which have harvested en masse from open source software (including GPL!) without permission -- to produce open source / free software.

Restoring a bit of balance to things.

Re: A guide to local coding models

#297

Post by Aurornis »

jgalt212 wrote: post #291
bilater wrote: post #289

If you are using local models for coding you are midwiting this. Your code should be worth more than a subscription. The only legit use case for local models is privacy. I don't know why anyone would want to code with an intern level model when they can get a senior engineer level model for a couple of bucks more. It DOESN'T MATTER if you're writing a simple hello world function or building out a complex feature. Jus…

I will use a local coding model for our proprietary / trade secrets internal code when Google uses Claude for its internal code and Microsoft starts using Gemini for internal code. The flip side of this coin is I'd be very excited if Jane Street or DE Shaw were running their trading models through Claude. Then I'd have access to billions of dollars of secrets.

> I'd be very excited if Jane Street or DE Shaw were running their trading models through Claude. Then I'd have access to billions of dollars of secrets.

Using Claude for inference does not mean the codebase gets pulled into their training set.

This is a tired myth that muddies up every conversation about LLMs

Re: A guide to local coding models

#298

Post by cmrdporcupine »

Lapel2742 wrote: post #224
hu3 wrote: post #135

Earlier quoted context omitted.

Open weight models are neat. But for SOTA performance you need specialized hardware. Even for Open Weight models. 40k in consumer hardware is never going to compete with 40k of AI specialized GPUs/servers. Your link starts with: > "Using a single top-of-the-line gaming GPU like NVIDIA’s RTX 5090 (under $2500), anyone can locally run models matching the absolute frontier of LLM performance from just 6 to 12 months ago…

> I highly doubt a RTX 5090 can run anything that competes with Sonnet 3.5 which was released June, 2024. I don't know about the capabilities of a 5090 but you probably can run a Devstral-2 [1] model locally on a Mac with good performance. Even the small Devstral-2 model (24b) seems to easily beat Sonnet 3.5 [2]. My impression is that local models have made huge progress. Coding aside I'm also impressed by the Minist…

Thing is you can pay basically fractions of cents a query to e.g. DeepSeek Platform or DeepInfra or Z.Ai or whatever and have them run the same open models for far cheaper and faster than you could ever build out at home.

It's neat to play with, but not practical.

The only story that I can see that makes sense for running at home is if you're going to fine tune a model by taking an open weight model and doing things to it and running that. Even then I believe there's places (hugging face?) that will host and run your updated model for cheaper than you could run it yourself.

Re: A guide to local coding models

#299

Post by Aurornis »

smcleod wrote: post #43
Workaccount2 wrote: post #12

I'm curious what the mental calculus was that a $5k laptop would competitively benchmark against SOTA models for the next 5 years was. Somewhat comically, the author seems to have made it about 2 days. Out of 1,825. I think the real story is the folly of fixating your eyes on shiny new hardware and searching for justifications. I'm too ashamed to admit how many times I've done that dance... Local models are purely fo…

My 2023 Macbook Pro (M2 Max) is coming up to 3 years old and I can run models locally that are arguably "better" than what was considered SOTA about 1.5 years ago. This is of course not an exact comparison but it's close enough to give some perspective.

I don't know about that. Even trying Devstral 2 locally feels less competent than the SOTA models from mid-2024.

It's impressive to see what I can run locally, but they're just not at the level of anything from the GPT-4 era in my experience.

Re: A guide to local coding models

#300

Post by Aurornis »

Lapel2742 wrote: post #224
hu3 wrote: post #135

Earlier quoted context omitted.

Open weight models are neat. But for SOTA performance you need specialized hardware. Even for Open Weight models. 40k in consumer hardware is never going to compete with 40k of AI specialized GPUs/servers. Your link starts with: > "Using a single top-of-the-line gaming GPU like NVIDIA’s RTX 5090 (under $2500), anyone can locally run models matching the absolute frontier of LLM performance from just 6 to 12 months ago…

> I highly doubt a RTX 5090 can run anything that competes with Sonnet 3.5 which was released June, 2024. I don't know about the capabilities of a 5090 but you probably can run a Devstral-2 [1] model locally on a Mac with good performance. Even the small Devstral-2 model (24b) seems to easily beat Sonnet 3.5 [2]. My impression is that local models have made huge progress. Coding aside I'm also impressed by the Minist…

> Even the small Devstral-2 model (24b) seems to easily beat Sonnet 3.5 [2].

I've played with Devstral 2 a lot since it came out. I've seen the benchmarks. I just don't believe it's actually better for coding.

It's amazing that it can do some light coding locally. I think it's great that we have that. But if I had to choose between a 2024-era model and Devstral 2 I'd pick the older Sonnet or GPTs any day.

Post reply on HN