[flagged]
AI bot
DeepSeek 4 Flash local inference engine for Metal
131–140 of 171 posts
Re: DeepSeek 4 Flash local inference engine for Metal
#132Re: DeepSeek 4 Flash local inference engine for Metal
#133Earlier quoted context omitted.
>It could run viably with SSD offload on Macs with very little memory Not really. That's going to land you somewhere in the 0.2-0.5 tokens a second range Lovely as modern nvmes are they're not memory
You can run multiple inferences in parallel on the same set of weights, that's what batching is. Given enough parallelization it can be almost entirely compute-limited, at least for small context (max ~10GB per request apparently, but that's for 1M tokens!)
Yes I think what this demonstrates that folks are missing is that now optimization for specific scenarios is quite possible.
Re: DeepSeek 4 Flash local inference engine for Metal
#134Heh, I made something very similar for the Qwen3 models a while back. It only runs Qwen3, supports only some quants, loads from GGUF, and has inference optimized by Claude (in a loop). The whole thing is compact (just a couple of files) and easy to reason about. I made it for my students so they could tinker with it and learn (add different decoding strategies, add abliteration, etc.). Popular frameworks are large, c…
Ultra-optimized HW-specific engines is what Mojo lang seems to be targeting, but I rarely hear about it here.
Re: DeepSeek 4 Flash local inference engine for Metal
#135Heh, I made something very similar for the Qwen3 models a while back. It only runs Qwen3, supports only some quants, loads from GGUF, and has inference optimized by Claude (in a loop). The whole thing is compact (just a couple of files) and easy to reason about. I made it for my students so they could tinker with it and learn (add different decoding strategies, add abliteration, etc.). Popular frameworks are large, c…
Ultra-optimized HW-specific engines is what Mojo lang seems to be targeting, but I rarely hear about it here.
> Mojo lang seems to be targeting, but I rarely hear about it here
Momentum over at Mojo lang seems very very slow.
According to their roadmap, they're still busy on Phase 1 ("High performance CPU + GPU coding"), and haven't touched Phase 2 ("Systems application programming") and Phase 3 ("Dynamic object-oriented programming").
So perhaps there isn't much to talk about?
Re: DeepSeek 4 Flash local inference engine for Metal
#136Earlier quoted context omitted.
> what if we started building ultra-optimized inference engines tailored to an exact GPU+model combination? The inference engines in use already include different backend building blocks optimized for different hardware. While there are places where you can pick up some low hanging fruit for less popular platforms, there isn't a lot of room to squeeze in super optimized model-runners for specific GPU families and get…
When you support multiple backends, you end up having to abstract over them. Each backend may implement the abstraction to the best of its capability, but you still have to deal with the abstraction sitting between your workload and its compute. Wouldn't it be nice if you didn't need that abstraction? That's what GP is talking about, I'm sure: optimizing the workload directly for the hardware, rather than merely the…
Absttaction doesnt always imply performance overhead.
Re: DeepSeek 4 Flash local inference engine for Metal
#137Earlier quoted context omitted.
>It could run viably with SSD offload on Macs with very little memory Not really. That's going to land you somewhere in the 0.2-0.5 tokens a second range Lovely as modern nvmes are they're not memory
You can run multiple inferences in parallel on the same set of weights, that's what batching is. Given enough parallelization it can be almost entirely compute-limited, at least for small context (max ~10GB per request apparently, but that's for 1M tokens!)
For offline work that's fine I guess, but batched or not <1tks is largely unusable for most usage cases
Re: DeepSeek 4 Flash local inference engine for Metal
#138Any direct TPS comparison to Ollama?
Re: DeepSeek 4 Flash local inference engine for Metal
#139Earlier quoted context omitted.
You can run multiple inferences in parallel on the same set of weights, that's what batching is. Given enough parallelization it can be almost entirely compute-limited, at least for small context (max ~10GB per request apparently, but that's for 1M tokens!)
For offline work that's fine I guess, but batched or not <1tks is largely unusable for most usage cases
I just think this potential workflow needs to be tested so that we know if anything breaks or makes it infeasible. Ultimately it would be slow when running any single agent, but you might be working with a huge amount of them in parallel. I view this as potentially a great way of repurposing low-RAM hardware with this specific model.
Re: DeepSeek 4 Flash local inference engine for Metal
#140Any direct TPS comparison to Ollama?
Ollama has no local support for DeepSeek V4 at present; it's only listed as a cloud model. Even llama.cpp is still waiting for support.