The path to open-sourcing the DeepSeek inference engine
1–10 of 69 posts
Re: The path to open-sourcing the DeepSeek inference engine
#2tl;dr "we had our vLLM fork and it's unmaintainable now; guess we are going to rebuild it, in the public this time"
Re: The path to open-sourcing the DeepSeek inference engine
#3tl;dr "we had our vLLM fork and it's unmaintainable now; guess we are going to rebuild it, in the public this time"
Unmaintainable seems unduly harsh. There is a big gap between maintainable internally and ready for public consumption
Re: The path to open-sourcing the DeepSeek inference engine
#4tl;dr "we had our vLLM fork and it's unmaintainable now; guess we are going to rebuild it, in the public this time"
I get the impression their setup is very hard to maintain but it's worth every penny. They've done optimizations that wring incredible performance out of the hardware they have, but they also have specific machine configurations and I wouldn't be surprised if they have complicated hacks that get 100% speedups for some stuff but those speedups disappear if you have a slightly different motherboard configuration. Also there's suggestion they've made firmware hacks which are worth it at their scale, but might be very dangerous and difficult to apply especially on a small scale. (And some of their hacks might involve both firmware and cluster-level optimizations, which would be useless or counterproductive independently.)
And even if you have somewhat similar hardware, the code might not be that helpful, you might be better off with a sketch of the solution and implementing it yourself. If you've got a large enough cluster it's going to pay for itself anyway.
Re: The path to open-sourcing the DeepSeek inference engine
#5tl;dr "we had our vLLM fork and it's unmaintainable now; guess we are going to rebuild it, in the public this time"
They're going to spend time and effort into making their optimizations public. Would you rather have them keep their changes internal?
Re: The path to open-sourcing the DeepSeek inference engine
#6Nice. We've seen some good engineering work from DeepSeek. Keep it coming.
Re: The path to open-sourcing the DeepSeek inference engine
#7I really empathised with this part:
> Codebase Divergence: Our engine is based on an early fork of vLLM from over a year ago. Although structurally similar, we’ve heavily customized it for DeepSeek models, making it difficult to extend for broader use cases.
I've been there. Probably a few of us have.
Their approach of working on splitting out maintainable sublibraries and sharing info directly even if not integrated seems a really nice way of working with the community -- ie, they have obstacles, but they're not letting the obstacles cause them to take the easy route of not contributing at all. And while it might seem better to someone wanting to use their techniques to share only working code, not info on the techniques, at least it's still knowledge sharing. And again I think it'd be easier for them not to do it. So kudos to them.
Re: The path to open-sourcing the DeepSeek inference engine
#8I really empathised with this part: > Codebase Divergence: Our engine is based on an early fork of vLLM from over a year ago. Although structurally similar, we’ve heavily customized it for DeepSeek models, making it difficult to extend for broader use cases. I've been there. Probably a few of us have. Their approach of working on splitting out maintainable sublibraries and sharing info directly even if not integrated…
They customized and optimized vLLM for their use case, so much that it became a different product (e.g. Debian vs Ubuntu).
The fact they share back some of their improvements is great.
Re: The path to open-sourcing the DeepSeek inference engine
#9In March, vLLM picked up some of the improvements in the DeepSeek paper. Through these, vLLM v0.7.3's DeepSeek performance jumped to about 3x+ of what it was before [1].
What's exciting is that there's still so much room for improvement. We benchmark around 5K total tokens/s with the sharegpt dataset and 12K total token/s with random 2000/100, using vLLM and under high concurrency.
DeepSeek-V3/R1 Inference System Overview [2] quotes "Each H800 node delivers an average throughput of 73.7k tokens/s input (including cache hits) during prefilling or 14.8k tokens/s output during decoding."
Yes, DeepSeek deploys a different inference architecture. But this goes onto show just how much room there is for improvement. Looking forward to more open source!
[1] https://developers.redhat.com/articles/2025/03/19/how-we-opt...
[2] https://github.com/deepseek-ai/open-infra-index/blob/main/20...
Re: The path to open-sourcing the DeepSeek inference engine
#10I really empathised with this part: > Codebase Divergence: Our engine is based on an early fork of vLLM from over a year ago. Although structurally similar, we’ve heavily customized it for DeepSeek models, making it difficult to extend for broader use cases. I've been there. Probably a few of us have. Their approach of working on splitting out maintainable sublibraries and sharing info directly even if not integrated…
Non-runnable code can be really useful. I often wish it was available for some papers even if I never run it just to check what they actually did, because text and equations are often not specific enough.