Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek 4 Flash local inference engine for Metal

github.com

Post reply on HN
Print view View original

51–60 of 171 posts

Re: DeepSeek 4 Flash local inference engine for Metal

#51

Post by dakolli »

afro88 wrote: post #12
dakolli wrote: post #11

Earlier quoted context omitted.

[flagged]

No offense, this is a crazy worthless contribution to the discussion. Why?

Because everyone in these replies is in complete denial about the physical limits of memory and scaling in general. Ya'll literally living in an alternate reality where model capability increases with a decrease in size, its simply not the case. There will be small focused models that preform well on very narrow tasks, yes, but you will not have "agents" capable of "building most things" running on consumer hardware until more capable (and affordable) consumer hardware exists.

Re: DeepSeek 4 Flash local inference engine for Metal

#52

Post by bensyverson »

dakolli wrote: post #51
afro88 wrote: post #12

Earlier quoted context omitted.

No offense, this is a crazy worthless contribution to the discussion. Why?

Because everyone in these replies is in complete denial about the physical limits of memory and scaling in general. Ya'll literally living in an alternate reality where model capability increases with a decrease in size, its simply not the case. There will be small focused models that preform well on very narrow tasks, yes, but you will not have "agents" capable of "building most things" running on consumer hardware…

Ah, you haven't realized that consumer hardware gets more capable over time

Re: DeepSeek 4 Flash local inference engine for Metal

#53

Post by mirsadm »

kgeist wrote: post #36

Heh, I made something very similar for the Qwen3 models a while back. It only runs Qwen3, supports only some quants, loads from GGUF, and has inference optimized by Claude (in a loop). The whole thing is compact (just a couple of files) and easy to reason about. I made it for my students so they could tinker with it and learn (add different decoding strategies, add abliteration, etc.). Popular frameworks are large, c…

I've built something like this. One issue is that LLMs are actually terrible at writing good shaders. I've spent way too much time trying to get them not to be so awful at it.

Re: DeepSeek 4 Flash local inference engine for Metal

#54

Post by cortesoft »

minimaxir wrote: post #17
antirez wrote: post #15

A random, funny, interesting and telling data point: my MacBook M3 Max while DS4 is generating tokens at full speed peaks 50W of energy usage...

"Data centers for LLMs are technically more energy efficient per-user than self-hosting LLM models due to economies-of-scale" is a data point the internet isn't ready for.

I thought this is a pretty generally accepted fact?

Re: DeepSeek 4 Flash local inference engine for Metal

#55

Post by j_maffe »

Lalabadie wrote: post #43
minimaxir wrote: post #17

Earlier quoted context omitted.

"Data centers for LLMs are technically more energy efficient per-user than self-hosting LLM models due to economies-of-scale" is a data point the internet isn't ready for.

Using only this dimension in a vacuum, it sounds like an easy choice, but we're extremely early in this market, and the big providers are already a mess of pricing choices, pricing changes, and sudden quota adjustments for consumers. Plus, a Mac that's not running inference idles down to 1-5W, only drawing power when it needs to. Datacenters must maximize usage, individuals and their devices don't have to. A Mac is a…

But it's simply an economic fact that EoS will be more efficient with a task that's so easy to offload somewhere else.

Re: DeepSeek 4 Flash local inference engine for Metal

#56

Post by xtracto »

kgeist wrote: post #36

Heh, I made something very similar for the Qwen3 models a while back. It only runs Qwen3, supports only some quants, loads from GGUF, and has inference optimized by Claude (in a loop). The whole thing is compact (just a couple of files) and easy to reason about. I made it for my students so they could tinker with it and learn (add different decoding strategies, add abliteration, etc.). Popular frameworks are large, c…

This takes me to the famous FizzBuzz High performance codegolf answer [1]. If we could implement optimizations like that for the inferences, maybe we could increase the speeds 10x or more.

[1] https://codegolf.stackexchange.com/questions/215216/high-thr...

Re: DeepSeek 4 Flash local inference engine for Metal

#57

Post by losvedir »

antirez wrote: post #15

A random, funny, interesting and telling data point: my MacBook M3 Max while DS4 is generating tokens at full speed peaks 50W of energy usage...

It's so interesting to think about how much power it takes these machines to "think". I think I had a vague notion that it was "a lot" but it's good to put a number on it.

If DS4 Flash peaks at 50W and is 280B parameters, does that mean DS4 Pro at 1.6T parameters would likely be 300W or so? And the latest GPT 5 and Opus which feel maybe comparable-ish around 500W? Is it fair to say that when I'm using Claude Code and it's "autofellating" or whatever I'm burning 500W in a datacenter somewhere during that time?

Re: DeepSeek 4 Flash local inference engine for Metal

#59

Post by actsasbuffoon »

unshavedyak wrote: post #22
amunozo wrote: post #13

I am curious about it producing less tokens except for the max mode. I love DeepSeek V4 Flash and I use it extensively, it's so cheap I can use it all day and still not use all my 10$ OpenCode Go subscription. I use it always in max mode because of this, but now I wonder whether I should rather use high.

What do you use it for? I tend to just stick to SOTA (Claude 4.7 Max thinking), and put up with the slow req/response. I'm not sure what type of work i'd trust a less thinking model, as my intuition is built around what Claude vSOTA Max can handle. Nonetheless eventually i want to build an at-home system. I imagine some smaller local model could handle metadata assignment quite well. edit : Though TIL Mac Studio does…

Apple just dropped the 128GB option as well.

Re: DeepSeek 4 Flash local inference engine for Metal

#60

Post by brcmthrowaway »

visarga wrote: post #37

Large LLMs on MacBook produce tokens at an acceptable speed but the problem is reading context. Not incremental reading like when you have a chat session, because they use KV cache, but large size reading, like when you paste a big file. It can take minutes.

Why is this the case?

Are there any architectures that don't rely on feeding the entire history back into the chat?

Recurrent LLMs?

Post reply on HN