Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek 4 Flash local inference engine for Metal

github.com

Post reply on HN
Print view View original

161–170 of 171 posts

Re: DeepSeek 4 Flash local inference engine for Metal

#161

Post by zozbot234 »

gghh wrote: post #160
zozbot234 wrote: post #159

Earlier quoted context omitted.

Keep in mind, I said serving many requests in parallel, not just many users . In fact it's even more efficient if you can batch the requests of a large subagent swarm in parallel since this allows for sharing a big chunk of context/KV cache not just the model weights. That's why I raised the possibility of leveraging this same efficiency with DeepSeek V4. If as a user I can get into the habit of just firing off a req…

Ah ok, sharing context/KV cache, I can see that helping. I need to learn more about DS V4, you seem to hint it has some advantages over previous generations in this respect. I haven't followed that closely to quite catch this argument, I'll check it out.

The basic argument is that its KV cache is roughly an order of magnitude more compact than previous Chinese models, which were already very compact compared to the likes of Gemma 4 (though that example is a bit of an extreme). If you pair this with the basic facts of how to maximize LLM inference performance at scale (this was recently talked about in a video lecture on the Dwarkesh Patel YouTube podcast) the case for doing slow batched inference on prem with DeepSeek V4, perhaps even with memory offload, becomes, as I see it, quite obvious. Of course, I'd like to be proven wrong!

Re: DeepSeek 4 Flash local inference engine for Metal

#162

Post by bensyverson »

zozbot234 wrote: post #157
bensyverson wrote: post #156

Earlier quoted context omitted.

If you're not paying attention to what's happening with small models, I suggest you take a closer look. Keeping parameter count constant, the quality of small models is rising fast. When you look at what you could do with Llama just 3 years ago vs Gemma 4 on the same 16GB hardware, the trend is clear. Meanwhile, this year Apple bumped the base of their Mac lineup from 8GB to 16GB RAM, and the iPhone 17 Pro ships with…

Small models are gaining useful reasoning ability and that's a genuinely helpful development, but they'll be heavily limited in world knowledge for the foreseeable future. BTW, the base of the Mac lineup is now once again a 8GB device with a small and low-performance SSD. Many people will tell you that it's broadly comparable (though of course not identical!) to the original base model M1.

For many tasks, including lots of agentic applications, world knowledge is not a "must-have."

To me the Neo is an exception, and doesn't represent the core Mac lineup, which is all at 16GB+ of RAM. If you're developing pro software that would rely on an on-device LLM, you probably wouldn't be targeting the Neo anyway.

Re: DeepSeek 4 Flash local inference engine for Metal

#163

Post by bensyverson »

wtallis wrote: post #104
fulafel wrote: post #103

Earlier quoted context omitted.

This seems to be a different discussion than was going on up thread about: > in the next few years a "good enough" model will run on entry-level hardware

Exactly. In the next few years , entry-level hardware will not be advancing beyond 16GB. And anything beyond 32GB will remain decidedly high-end. And that's for laptops with unified memory. In the desktop space, 8GB discrete GPUs are going to be sticking around for a very long time.

I guess we'll find out! I bet all the vendors who supply RAM are looking at the current shortages and thinking "well, it's a shame we could never manufacture more RAM than we currently do."

Re: DeepSeek 4 Flash local inference engine for Metal

#164

Post by gghh »

zozbot234 wrote: post #161
gghh wrote: post #160

Earlier quoted context omitted.

Ah ok, sharing context/KV cache, I can see that helping. I need to learn more about DS V4, you seem to hint it has some advantages over previous generations in this respect. I haven't followed that closely to quite catch this argument, I'll check it out.

The basic argument is that its KV cache is roughly an order of magnitude more compact than previous Chinese models, which were already very compact compared to the likes of Gemma 4 (though that example is a bit of an extreme). If you pair this with the basic facts of how to maximize LLM inference performance at scale (this was recently talked about in a video lecture on the Dwarkesh Patel YouTube podcast) the case fo…

Right, Dwarkesh's episode with Reiner Pope. Didn't watch the full video but as soon I saw both going to an old school blackboard with an actual chalk in hand I could tell they meant business hehe :) Thanks for recommending the vid and for the info about DS V4.

Re: DeepSeek 4 Flash local inference engine for Metal

#165

Post by tmzt »

lhl wrote: post #93

I think especially with the ability for SOTA AI to optimize kernels more people should try their hand at making better inference for their specific hardware. I have an older W7900 (RDNA3) which, besides 48GB of VRAM, has some pretty decent roofline specs - 123 FP16 TFLOPS/INT8 TOPS, 864 GB/s MBW, but has had notoriously bad support both from AMD (ROCm) as well as llama.cpp. Recently I decided I'd like to turn the car…

Doing the same for Apple M-series with fused wgsl shaders specifically targeting Qwen3/3.5.

My effort is called shady-thinker and is on github at github.com/tmzt/shady-thinker.

This was inspired in part by Antirez's earlier work with C kernels as well as other efforts to support in-browser LLMs. I've adapted them to Rust and the wgpu library.

Gemma 4 is also the next likely target (with the MTP work) as I'm experimenting with local AI agents.

I'd love to see what you've done to improve prefill and decode even if its not directly applicable.

One difference, I'm using MLX and GPTQ 4bit quants including AutoRound with safetensors as my shader pipeline is pretty much fixed for each model, ggml just adds unnecessary complexity.

Re: DeepSeek 4 Flash local inference engine for Metal

#167

Post by aeonfox »

ifeot wrote: post #116
aeonfox wrote: post #90

Earlier quoted context omitted.

Separate to the self-host/datacentre argument, it would be interesting to see a speed/performance/watts-per-token leaderboard between leading models. Which model is the most watt-efficient?

Akbaruddin

?

Re: DeepSeek 4 Flash local inference engine for Metal

#168

Post by dev_hugepages »

jiehong wrote: post #41
speu wrote: post #5

I've been trying deepseek-v4-flash in OpenCode (via OpenRouter) and I'm blown away. It's no Opus, obviously, but it had zero issues with any regular coding task I threw at it. v4-flash is remarkably "good enough" for what I needed. The whole evening of coding cost me $0.52 in API credits.

Using it in Kagi Assistant is stupidly slow. I get like 10 t/s. While it’s pretty fast in the official app for example. Kagi Assistant is also kind of broken when using Qwen 3.6 Plus. So, beware of using them in Kagi at the moment.

Probably a provider thing. Looking at https://help.kagi.com/kagi/ai/llms-privacy.html, they're using deepinfra.

Looking at https://openrouter.ai/deepseek/deepseek-v4-flash/providers tells us that the deepseek provider achieves 49tps of throughput while deepinfra 19tps.

Re: DeepSeek 4 Flash local inference engine for Metal

#169

Post by jiehong »

dev_hugepages wrote: post #168
jiehong wrote: post #41

Earlier quoted context omitted.

Using it in Kagi Assistant is stupidly slow. I get like 10 t/s. While it’s pretty fast in the official app for example. Kagi Assistant is also kind of broken when using Qwen 3.6 Plus. So, beware of using them in Kagi at the moment.

Probably a provider thing. Looking at https://help.kagi.com/kagi/ai/llms-privacy.html , they're using deepinfra. Looking at https://openrouter.ai/deepseek/deepseek-v4-flash/providers tells us that the deepseek provider achieves 49tps of throughput while deepinfra 19tps.

Thanks for taking the time to provide this info. I appreciate it

Re: DeepSeek 4 Flash local inference engine for Metal

#170

Post by lhl »

throwa356262 wrote: post #113
lhl wrote: post #93

I think especially with the ability for SOTA AI to optimize kernels more people should try their hand at making better inference for their specific hardware. I have an older W7900 (RDNA3) which, besides 48GB of VRAM, has some pretty decent roofline specs - 123 FP16 TFLOPS/INT8 TOPS, 864 GB/s MBW, but has had notoriously bad support both from AMD (ROCm) as well as llama.cpp. Recently I decided I'd like to turn the car…

Please share your knowledge and your findings I think llama.cpp could have done a much better job supporting PC. Sure, some of it us due to bad vendor support but with so many users I am surprised we don't see more optimized inference on standard PCs

Took a little longer to clean up than I expected. I'd recommend checking out the ROOFLINE and the LESSONS-LEARNED docs here: https://github.com/shisa-ai/hipEngine/tree/main/docs
Post reply on HN