Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models on an M4 with 24GB memory

jola.dev

Post reply on HN
Print view View original

161–170 of 189 posts

Re: Running local models on an M4 with 24GB memory

#161

Post by OtomotO »

solenoid0937 wrote: post #41
quacker wrote: post #37

I could have used this article before I spent the weekend arriving to the same conclusion! Same laptop, and my contrived test was having it fix 50 or so lint errors in a small vibe-coded C++ repo. I wanted it to be able to handle a bunch of small tasks without getting stuck too often. GPT OSS 20B was usable but slow, and actually frequently made mistakes like adding or duplicating statements unnecessarily, listing th…

> It is absolutely not comparable to frontier models. This is not said often enough. Yes, local LLMs are great! But reading most HN posts on the subject, you'd think they're within reach of Opus 4.7. There is a very small, very vocal, very passionate crowd that dramatically overstates the capabilities of local LLMs on HN.

That's totally fine and dandy as there is a very big, very vocal, very brainwashed crowd that dramatically overstates the capabilities of remote LLMs on HN as well.

Re: Running local models on an M4 with 24GB memory

#163

Post by losvedir »

claysmithr wrote: post #153
isaisabella wrote: post #75

I'd rather spend thousands dollars on a Mac than subscribing API. The local model allows me to do my work any time and anywhere, without worrying about privacy leak.

me too. plus, I don't like the idea of needing massive datacenters, it's not good for anybody

Datacenters are more efficient, though, because of batching.

Re: Running local models on an M4 with 24GB memory

#165

Post by onlyrealcuzzo »

ChrisMarshallNY wrote: post #139

> The longer you let it drive without constraints, the worse the wreckage gets. The velocity makes you think you're winning right up until the moment everything collapses simultaneously. In my experience (so far), I can’t let the LLM write too much in one go. I need to test the hell out of what it gives me, and I can’t ask for too much, at one time. I tend to ask it to “flesh out” functions, where I have a signature,…

> In my experience (so far), I can’t let the LLM write too much in one go.

Second, but I've found a cheat code to make it much farther with minimal intervention.

Step 1: tell them your goal, have them generate a doc, include design principals, system invariants, and acceptance criteria.

No amount of CLAUDE.md or skills beats re-iterating the focus points directly in the prompt.

Step 2: tell them to summarize the doc (pay close attention here). Have them save it somewhere (I use docs/agents) once you're happy with it.

Step 3: tell them to build a detailed plan to meet the objectives of the doc.

Step 4: let them go wild.

Step 5: once they declare "done", feed their progress to another LLM (Gemini is quite decent for review, and free) -> mindlessly feed the feedback back to the implementing LLM.

Step 6: Say the magic words: https://github.com/cuzzo/clear/blob/master/docs/retrospectiv...

Again, I've found no amount of skills or CLAUDE.md beats slightly modifying a prompt to meet your exact goals specific to the design and what you know of the implementation so far.

Step 7: Have them rebuild a plan to address feedback.

Step 8: Let them go wild. Loop back to Step 5 until the LLMs tell you there's no major action items.

Step 9: Tell them to remove anything from the commit that's not strictly necessary, get rid of comment changes that aren't strictly necessary, etc.

Step 10: here and only here do you invest your time (worth 100x what you're paying them) to look at what they did. Here you can give them feedback to address anything you saw.

Step 11: Review.

Step 12: Profit $$$

I got a quite decent implementation of Finite State Machine and Thunk + Trampoline transformation of code in custom language I'm building in about 1 day, barely checking in while commuting to and from work on the train...

Occassionally, at step 11, you will find a gigantic turd and wonder how the LLMs converged on this. But, typically, it's at least good enough at that stage.

I don't even waste my time looking at anything they've done until they've converged on a good design and implementation with no holes, no feedback, no notes that does what a minimal, summarized doc clearly states and follows the design principles. Because they DEFINITELY haven't in a one-shot.

Re: Running local models on an M4 with 24GB memory

#167

Post by adamsb6 »

I realized that should I end up getting laid off soon I won't have an unlimited token budget and for the workflows I've settled into it would be quite expensive. So I was exploring what it would take to run open models at home.

Was quite disappointed to see that the PC side hasn't kept up. The unified architecture on Macs makes it very hard to justify spending money on a Linux machine for inference workloads.

Re: Running local models on an M4 with 24GB memory

#168

Post by ChrisMarshallNY »

OtomotO wrote: post #160
ChrisMarshallNY wrote: post #139

> The longer you let it drive without constraints, the worse the wreckage gets. The velocity makes you think you're winning right up until the moment everything collapses simultaneously. In my experience (so far), I can’t let the LLM write too much in one go. I need to test the hell out of what it gives me, and I can’t ask for too much, at one time. I tend to ask it to “flesh out” functions, where I have a signature,…

No, there are millions upon millions of mediocre lines of code out there. And LLMs tend to converge on mediocrity. Which is totally fine.

I find that there’s a surprising amount of really good stuff out there.

PHP has come of age. Actually, it’s been a backbone technology for millions of professional sites and apps for many years, and people tend to work in the open. Sort of the nature of the language.

There’s a popular perception that PHP programmers are bad programmers, but that’s a dated point of view. Pros have been using it to make serious money, and create serious infrastructure, for many years.

Re: Running local models on an M4 with 24GB memory

#169

Post by neverrroot »

A few interesting undertone points in the article, for those who care:

- reliance on US technologies is not so good, but on Chinese is not discussed, just chosen

- environmental cost is of concern

- so are the energy costs

In the end, there are some clear tips on how to configure the LLM, but overall the article is a bit thin and rather biased.

Re: Running local models on an M4 with 24GB memory

#170

Post by BubbleRings »

solenoid0937 wrote: post #44
BubbleRings wrote: post #40

Earlier quoted context omitted.

Yeah no kidding. For instance, if you are an independent inventor trying to write a patent while keeping your patent lawyer expenses to a minimum, you want to write as much of the first draft(s) of the patent as you can yourself. (You’ll save billable hours with your patent lawyer, and you’ll end up with a better patent because you’ll communicate your innovations more clearly to your lawyer.) However, and this is the…

> This is because describing your invention to a web based LLM could be considered a public “disclosure” of your invention, which, (after a one year grace period goes by), could put your invention in the public domain, basically—and thereby prevent you (or anyone else) from being able to ever patent the invention. This is simply not true. Even if it were true (and again, it's not) you could simply use zero data reten…

If a competitor to your business discovers that you used a free online AI to help draft your patent 1.5 years ago, that competitor could then cause your patent to be invalidated, which could be greatly to their benefit of course.

The Terms of Service (ToS) for Open/Public AI (e.g., free consumer versions of ChatGPT, Gemini, Claude) often reserve the right to store your prompts and use them to train and refine the model.

Doing an enabling disclosure of your patent draft to another party that is not bound by a non-disclosure agreement is a big mistake, at least while the case law has not yet been settled.

My post was meant to be encouraging to people that might be considering local LLM for this specific use case, where protecting confidential information is of particular importance.

Post reply on HN