Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek 4 Flash local inference engine for Metal

github.com

Post reply on HN
Print view View original

41–50 of 171 posts

Re: DeepSeek 4 Flash local inference engine for Metal

#41

Post by jiehong »

speu wrote: post #5

I've been trying deepseek-v4-flash in OpenCode (via OpenRouter) and I'm blown away. It's no Opus, obviously, but it had zero issues with any regular coding task I threw at it. v4-flash is remarkably "good enough" for what I needed. The whole evening of coding cost me $0.52 in API credits.

Using it in Kagi Assistant is stupidly slow. I get like 10 t/s.

While it’s pretty fast in the official app for example.

Kagi Assistant is also kind of broken when using Qwen 3.6 Plus.

So, beware of using them in Kagi at the moment.

Re: DeepSeek 4 Flash local inference engine for Metal

#42

Post by bel8 »

visarga wrote: post #37

Large LLMs on MacBook produce tokens at an acceptable speed but the problem is reading context. Not incremental reading like when you have a chat session, because they use KV cache, but large size reading, like when you paste a big file. It can take minutes.

And unless I'm mistaken, the repo is about running it with 2bit quantization.

This is probably far from the raw intelligence provided by cloud providers.

Still, this shines more light on local LLMs for agentic workflows.

Re: DeepSeek 4 Flash local inference engine for Metal

#43

Post by Lalabadie »

minimaxir wrote: post #17
antirez wrote: post #15

A random, funny, interesting and telling data point: my MacBook M3 Max while DS4 is generating tokens at full speed peaks 50W of energy usage...

"Data centers for LLMs are technically more energy efficient per-user than self-hosting LLM models due to economies-of-scale" is a data point the internet isn't ready for.

Using only this dimension in a vacuum, it sounds like an easy choice, but we're extremely early in this market, and the big providers are already a mess of pricing choices, pricing changes, and sudden quota adjustments for consumers.

Plus, a Mac that's not running inference idles down to 1-5W, only drawing power when it needs to. Datacenters must maximize usage, individuals and their devices don't have to.

A Mac is also the rest of the personal computer!

Re: DeepSeek 4 Flash local inference engine for Metal

#44

Post by joshmarlow »

kgeist wrote: post #36

Heh, I made something very similar for the Qwen3 models a while back. It only runs Qwen3, supports only some quants, loads from GGUF, and has inference optimized by Claude (in a loop). The whole thing is compact (just a couple of files) and easy to reason about. I made it for my students so they could tinker with it and learn (add different decoding strategies, add abliteration, etc.). Popular frameworks are large, c…

Another suggestion for optimizing local inference - the Hermes team talks a lot on X about how much better results are when you use custom parsers tuned to the nuances of each model. Some models might like to use a trailing `,` in JSON output, some don't - so if your parser can handle the quirks of the specific model, then you get higher-performing functionality.

Re: DeepSeek 4 Flash local inference engine for Metal

#45

Post by dakolli »

amunozo wrote: post #14
dakolli wrote: post #6

Earlier quoted context omitted.

There will always be a huge gap between frontier models and open source models (unless you're very rich). This whole industry makes no sense, everyone is ignoring the unit economics. It cost 20k a month to running Kimi 2.6 at decent tok/ps, to sell those tokens at a profit you'd need your hardware costs to be less 1k a month. Everyone who's betting their competency on the generosity of billionaires selling tokens for…

Most tasks do not require frontier models, so as long as these models cover 95-99 per cent of the tasks, closed frontier models can be left for niche and specialized cases that are harder.

Frontier models can hardly do the tasks I want them too, I simply cannot buy into this notion.

Re: DeepSeek 4 Flash local inference engine for Metal

#46

Post by jwr »

antirez wrote: post #15

A random, funny, interesting and telling data point: my MacBook M3 Max while DS4 is generating tokens at full speed peaks 50W of energy usage...

Not everybody might realize this, but this is a truly excellent and very impressive result. Most models on my M4 Max run at 150W consumption.

Re: DeepSeek 4 Flash local inference engine for Metal

#47

Post by amunozo »

unshavedyak wrote: post #22
amunozo wrote: post #13

I am curious about it producing less tokens except for the max mode. I love DeepSeek V4 Flash and I use it extensively, it's so cheap I can use it all day and still not use all my 10$ OpenCode Go subscription. I use it always in max mode because of this, but now I wonder whether I should rather use high.

What do you use it for? I tend to just stick to SOTA (Claude 4.7 Max thinking), and put up with the slow req/response. I'm not sure what type of work i'd trust a less thinking model, as my intuition is built around what Claude vSOTA Max can handle. Nonetheless eventually i want to build an at-home system. I imagine some smaller local model could handle metadata assignment quite well. edit : Though TIL Mac Studio does…

I am experimenting with some game development and my thesis' beamer. I have a 20$ Codex account and I use GPT-5.5 for planning and DeepSeek for executing in OpenCode. This makes my Codex 5h tokens to last more than 10 minutes.

Re: DeepSeek 4 Flash local inference engine for Metal

#48

Post by amunozo »

syntaxing wrote: post #26
amunozo wrote: post #13

I am curious about it producing less tokens except for the max mode. I love DeepSeek V4 Flash and I use it extensively, it's so cheap I can use it all day and still not use all my 10$ OpenCode Go subscription. I use it always in max mode because of this, but now I wonder whether I should rather use high.

How has opencode go been for you? Worth changing over from Claude pro?

Given the price, extremely satisfied, especially thanks to DeepSeek V4 Flash that makes it last forever. I use it on top of my 20$ Codex which is great but tokens last nothing.

Re: DeepSeek 4 Flash local inference engine for Metal

#49

Post by antirez »

visarga wrote: post #37

Large LLMs on MacBook produce tokens at an acceptable speed but the problem is reading context. Not incremental reading like when you have a chat session, because they use KV cache, but large size reading, like when you paste a big file. It can take minutes.

DS4 can process 460 prompt tokens per second. Not stellar but not so slow. On M3 max. See the benchmarks on readme.

Re: DeepSeek 4 Flash local inference engine for Metal

#50

Post by amunozo »

PhilippGille wrote: post #34
amunozo wrote: post #13

I am curious about it producing less tokens except for the max mode. I love DeepSeek V4 Flash and I use it extensively, it's so cheap I can use it all day and still not use all my 10$ OpenCode Go subscription. I use it always in max mode because of this, but now I wonder whether I should rather use high.

On max it uses more than twice as many tokens as on high when running the ArtificialAnalysis benchmark suite, and then it's indeed the model with the highest token usage (among the current top tier models). See the "Intelligence vs. Token Use" chart here: https://artificialanalysis.ai/models?models=gpt-5-5%2Cgpt-5-...

Wow, the difference is quite considerable and the gain in intelligence is not that much. I might try to use high and just iterate more often. I am working with hobby stuff so I don't have to worry whether it breaks things or not.
Post reply on HN