Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models on an M4 with 24GB memory

jola.dev

Post reply on HN
Print view View original

171–180 of 189 posts

Re: Running local models on an M4 with 24GB memory

#171

Post by Melatonic »

pdyc wrote: post #127
2ndorderthought wrote: post #102

Earlier quoted context omitted.

Out of curiosity have you tried other small models? The e2b for me was unusable. Llama3.2 3b was better and that thing is a year old and I rarely use it now too.

yes i keep on trying small models, i have also tried qwen 3.5 0.8B, 2B, 4b and gemma4 e4B models but they either did not worked reliably (thinking loop, issue in following instruction) or there were performance issues (prompt speed, tg speed, too much ram) e2b was the sweet spot where i could give it plan and it can edit files properly.

How did e2b compare to e4b ?

Re: Running local models on an M4 with 24GB memory

#172

Post by hugmynutus »

krferriter wrote: post #159
hugmynutus wrote: post #146

Earlier quoted context omitted.

> What it gives me in Swift, most closely resembles stuff that enthusiastic newer folks would do, and want to show off. The same is true for rust-lang. Code that will immediately clone/re-allocate anything passed by reference and collect everything to the heap that is passed by `Iterator`/`IntoIterator`. It is a massive performance anti-pattern and the hallmark of somebody "struggling" with the borrow checker. Natura…

> It has gotten to a point that all code I generate with Opus or Codex if there as iterator or reference in the argument, I start a fresh context, with a sort of `remove unnecessary clones, collections, and copies from the following code: {{code}}` What does it do if you put "Avoid unnecessary clones, collections, and copies" in your CLAUDE.md/AGENTS.md?

It makes no difference at all.

Edit: Opus prior to the context nerf it worked more often than not. Current Opus 4.7 is practically unusable.

Re: Running local models on an M4 with 24GB memory

#173

Post by pdyc »

Melatonic wrote: post #171
pdyc wrote: post #127

Earlier quoted context omitted.

yes i keep on trying small models, i have also tried qwen 3.5 0.8B, 2B, 4b and gemma4 e4B models but they either did not worked reliably (thinking loop, issue in following instruction) or there were performance issues (prompt speed, tg speed, too much ram) e2b was the sweet spot where i could give it plan and it can edit files properly.

How did e2b compare to e4b ?

i did not see much improvement for my use case i.e. file editing tasks but with e4b tg/s is lower so i stick with e2b.

Re: Running local models on an M4 with 24GB memory

#176

Post by OtomotO »

ChrisMarshallNY wrote: post #168
OtomotO wrote: post #160

Earlier quoted context omitted.

No, there are millions upon millions of mediocre lines of code out there. And LLMs tend to converge on mediocrity. Which is totally fine.

I find that there’s a surprising amount of really good stuff out there. PHP has come of age. Actually, it’s been a backbone technology for millions of professional sites and apps for many years, and people tend to work in the open. Sort of the nature of the language. There’s a popular perception that PHP programmers are bad programmers, but that’s a dated point of view. Pros have been using it to make serious money,…

I am not dissing PHP, I am saying that the absolute majority of any code out there is mediocre and not super good and not super bad.

Re: Running local models on an M4 with 24GB memory

#177

Post by solenoid0937 »

BubbleRings wrote: post #170
solenoid0937 wrote: post #44

Earlier quoted context omitted.

> This is because describing your invention to a web based LLM could be considered a public “disclosure” of your invention, which, (after a one year grace period goes by), could put your invention in the public domain, basically—and thereby prevent you (or anyone else) from being able to ever patent the invention. This is simply not true. Even if it were true (and again, it's not) you could simply use zero data reten…

If a competitor to your business discovers that you used a free online AI to help draft your patent 1.5 years ago, that competitor could then cause your patent to be invalidated, which could be greatly to their benefit of course. The Terms of Service (ToS) for Open/Public AI (e.g., free consumer versions of ChatGPT, Gemini, Claude) often reserve the right to store your prompts and use them to train and refine the mod…

1. This absolutely wouldn't count as disclosure under the eye of the law.

2. ZDR + frontier LLMs would still be far more effective than local LLMs.

3. By your logic you can't upload patent drafts to Google Docs because Google hasn't signed an NDA, but this obviously is not the case.

The law is a lot less strict to the letter than you think it is. Intent and spirit of the law matter. Any judge would throw such a case out.

Re: Running local models on an M4 with 24GB memory

#178

Post by ChrisMarshallNY »

OtomotO wrote: post #176
ChrisMarshallNY wrote: post #168

Earlier quoted context omitted.

I find that there’s a surprising amount of really good stuff out there. PHP has come of age. Actually, it’s been a backbone technology for millions of professional sites and apps for many years, and people tend to work in the open. Sort of the nature of the language. There’s a popular perception that PHP programmers are bad programmers, but that’s a dated point of view. Pros have been using it to make serious money,…

I am not dissing PHP, I am saying that the absolute majority of any code out there is mediocre and not super good and not super bad.

I get the feeling that there’s a lot more “mediocre-to-good” PHP code (probably C, as well), than Swift.

I see terrible Swift code, frequently. It’s the kind of language that encourages “clever” approaches (as do most new languages), and these are favorites of less-experienced devs.

Re: Running local models on an M4 with 24GB memory

#179

Post by latable »

thot_experiment wrote: post #158
K0balt wrote: post #117

Earlier quoted context omitted.

What is your opinion on qwen 35b MOEvs qwen 27b dense?

Maybe a skill issue but they both feel about the same and the MoE is 3x faster so I barely use the dense model.

Not the person asked but on a medium bug that would span a few python files, I found the MOE be too enthusiastic trying things without trying to understand first the issue, when the dense model though hard and added debug statements to understand how to fix it. But the dense model is quite slow (Q4KM quant, MI50 32GB, llama.cpp, pi)

Re: Running local models on an M4 with 24GB memory

#180

Post by fg137 »

2ndorderthought wrote: post #144
solenoid0937 wrote: post #140

Earlier quoted context omitted.

It's just not there yet. I have tried all the models from April, including the Gemma 4 variants. These are so far from Opus it's not even funny. They are not close to being in the same league. Gemma might be like a frontier model from a couple years ago, but with much worse performance in long context chats.

Correct they aren't opus. They are sonnet with a little hand holding. They also run on a single GPU at 40 tps. No one is saying a local model will give you anthropics business in a 5min download. People are saying, "hmm, maybe I should do this one locally". People are also saying "this is surprisingly good enough for me given the trade offs"

> "hmm, maybe I should do this one locally"

If your time is worth nothing to even triage that question.

Unless you have fanatic needs for data privacy or really don't have Internet, running local models almost certainly results in negative ROI overall.

Not to mention that you need to have decent hardware (that is getting expensive by the day) to even have this conversation in the first place.

People in this post talk as if everyone has a Mac with 24GB or 32GB RAM. When the reality is that most people use a Windows laptop with crappy integrated GPU.

Post reply on HN