vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep
1–10 of 61 posts
Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep
#2Impressive performance work. It's interesting that you still see these 40+% perf gains like this.
Makes you think that you will continue to see the costs for a fixed level of "intelligence" dropping.
Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep
#3Love vLLM!
Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep
#4Now all we need is better support for AMD gpus, both CDNA and RDNA types
Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep
#5As a user of a lot of coding tokens I’m most interested in latency - these numbers are presumably for heavily batched workloads. I dearly wish Claude had a cerebras endpoint.
I’m sure I’d use more tokens because I’d get more revs, but I don’t think token usage would increase linearly with speed: I need time to think about what I want to and what’s happened or is proposed. But I feel like I would be able to stay in flow state if the responses were faster, and that’s super appealing.
Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep
#6Impressive performance work. It's interesting that you still see these 40+% perf gains like this. Makes you think that you will continue to see the costs for a fixed level of "intelligence" dropping.
Absolutely. LLM inference is still a greenfield — things like overlap scheduling and JIT CUDA kernels are very recent. We’re just getting started optimizing for modern LLM architectures, so cost/perf will keep improving fast.
Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep
#7Now all we need is better support for AMD gpus, both CDNA and RDNA types
ZLUDA implements CUDA on top of AMD ROCm - they are explicitly targetting vLLM as their PyTorch compatibility test: https://vosen.github.io/ZLUDA/blog/zluda-update-q4-2025/#pyt...
(PyTorch does also support ROCm generally, it shows up as a CUDA device.)
Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep
#8Still have to update it for snakepit 0.11.0, but I did start a vLLM wrapper for Elixir
Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep
#9I couldn't care less, tbh. This speed is ridiculously high, to the point where tool calls, not inference, become the bottleneck.
Also, I'd rather run a large model at slower speeds than a smaller at insanely high speeds.
Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep
#10Now all we need is better support for AMD gpus, both CDNA and RDNA types
ZLUDA implements CUDA on top of AMD ROCm - they are explicitly targetting vLLM as their PyTorch compatibility test: https://vosen.github.io/ZLUDA/blog/zluda-update-q4-2025/#pyt... (PyTorch does also support ROCm generally, it shows up as a CUDA device.)
I feel like these technologies are named by the Polish at the companies. "CUDA" means "WONDERS" and "ZŁUDA" would be an "ILLUSION".