Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

blog.vllm.ai

Post reply on HN
Print view View original

41–50 of 61 posts

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#41

Post by supermatt »

menaerus wrote: post #37
supermatt wrote: post #35

Earlier quoted context omitted.

I am in Lithuania, which has one of the highest wholesale energy prices in Europe (as per nord pool): https://data.nordpoolgroup.com/auction/day-ahead/prices?deli... That it is not translating into a higher cost to the consumer (as evidenced on your link) is likely indicative of other costs being incurred by the “average” consumer in those countries with a higher domestic rate - like massive markup from users being t…

Some countries also employ progressive electricity pricing such that higher energy consumption leads to elevated kWh rates incentivizing conservation. This is also not visible in the stats above. I also think that business kWh rates are actually higher than for the households in some instances.

Yeah, strictly business vs residential isn’t a good comparison either really, as the lower transmission fees for medium (10kV+) and higher voltage are where a lot of the savings are - and obv a lot of business don’t use such power.

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#42

Post by dust42 »

menaerus wrote: post #33
dust42 wrote: post #16

If I followed the links correctly this benchmark was made on a 16xH200. At current prices I'd assume that is a system price of around $750,000. The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages. Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated wi…

How did you arrive to $10,000 electricity costs figure? 8xH200 enclosed in DGX H200 system power draw is ~14kW in its peak (CTS) configuration/utilization. Over one year, and assuming maximum utilization, this is 123,480 kWh per single DGX H200 unit. We need 2x such units for 16xH200 system configuration under subject so it's 246,960 kWh/year. This is ~$25,000 at 10cts per kWh and ~$74,000 at 30cts per kWh. At ~1,110…

I used 700W per H200 = 11.2 per 16 GPUs. I didn't include CPU and rest of the rack. So yours is a better approximation.

One has to keep in mind that the benchmark that was done is synthetic. This makes sense because it makes it reproducible but real world usage may differ - i.e. by the amount of context and the number of concurrent users. Also there are use cases where smaller models or smaller quants will do.

The key take away for me for this type of back of the envelope calculation is to get a good idea where we stand long term, i.e. when VC money stops subsidizing.

So for me $0.3 per 1M tokens for a decent model looks pretty good too. Seeing that OpenAI API charges $21 per 1M tokens input and $168 output for GPT-5.2 pro I was wondering what the real sustainable pricing is.

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#43

Post by glemion43 »

supermatt wrote: post #25
dust42 wrote: post #16

If I followed the links correctly this benchmark was made on a 16xH200. At current prices I'd assume that is a system price of around $750,000. The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages. Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated wi…

> more like 30cts like it is in Europe Nope - i live in one of the most expensive areas, and even the residential price has averaged 18c/kWh delivered including taxes. Businesses get a lower basic rate and also don't pay the VAT, so it works out around 13c/kWh for them. https://data.nordpoolgroup.com/auction/day-ahead/prices?deli...

Private / endconsumer in Germany is 34

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#44

Post by Gracana »

ikari_pl wrote: post #10
mappu wrote: post #7

Earlier quoted context omitted.

ZLUDA implements CUDA on top of AMD ROCm - they are explicitly targetting vLLM as their PyTorch compatibility test: https://vosen.github.io/ZLUDA/blog/zluda-update-q4-2025/#pyt... (PyTorch does also support ROCm generally, it shows up as a CUDA device.)

I feel like these technologies are named by the Polish at the companies. "CUDA" means "WONDERS" and "ZŁUDA" would be an "ILLUSION".

ZLUDA was definitely intentional: https://github.com/vosen/ZLUDA/discussions/192

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#45

Post by [deleted] »

glemion43 wrote: post #43
supermatt wrote: post #25

Earlier quoted context omitted.

> more like 30cts like it is in Europe Nope - i live in one of the most expensive areas, and even the residential price has averaged 18c/kWh delivered including taxes. Businesses get a lower basic rate and also don't pay the VAT, so it works out around 13c/kWh for them. https://data.nordpoolgroup.com/auction/day-ahead/prices?deli...

Private / endconsumer in Germany is 34

[deleted]

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#46

Post by sofixa »

bildung wrote: post #34
sofixa wrote: post #23

Earlier quoted context omitted.

You can run vLLM with AMD GPUs supported by ROCm: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/infer... However from experience with an AMD Strix Halo, a couple of caveats: it's drastically slower than Ollama (tested over a few weeks, always using the official AMD vLLM nightly releases), and not all GPUs were supported for all models (but that has been fixed).

vLLM ususally only plays out its strength when serving multiple users in parallel, in contrast to llama.cpp (Ollama is a wrapper around llama.cpp). If you want more performance, you could try running llama.cpp directly or use the prebuilt lemonade nightlies.

But vLLM was half the t/s of Ollama, so something was obviously not ok.

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#47

Post by aurareturn »

yorwba wrote: post #18
dust42 wrote: post #16

If I followed the links correctly this benchmark was made on a 16xH200. At current prices I'd assume that is a system price of around $750,000. The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages. Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated wi…

It's 2.2k tokens per second and GPU, so you have to multiply the token output by 16 and the price per million tokens works out to 22.5 cents.

I think they're also running this at 16 bit quant. If they lower it to 8bit, they might double their output which might come out to be 11 cents per million tokens.

Now take into account that modern LLMs tend to use 4bit inference, and Blackwell is significantly more optimized for 4 bit, we can see much less than 11 cents. Maybe a speed up of 5x if using 4bit and Blackwell vs H100 and 8 bit?

So we're looking at potentially 2.2 cents per million tokens.

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#48

Post by aurareturn »

mycelia wrote: post #26

Hey HN! I’m Seiji Eicher from Anyscale, one of the authors of this post :) Feel free to ask questions here.

Are you using 16bit for inference? How many tokens/second if you use 8bit?

Given that SOTA models now use 4bit inference, can you do an estimation for 4bit + Blackwell?

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#49

Post by [deleted] »

dust42 wrote: post #16

If I followed the links correctly this benchmark was made on a 16xH200. At current prices I'd assume that is a system price of around $750,000. The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages. Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated wi…

[deleted]

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#50

Post by [deleted] »

aurareturn wrote: post #47
yorwba wrote: post #18

Earlier quoted context omitted.

It's 2.2k tokens per second and GPU, so you have to multiply the token output by 16 and the price per million tokens works out to 22.5 cents.

I think they're also running this at 16 bit quant. If they lower it to 8bit, they might double their output which might come out to be 11 cents per million tokens. Now take into account that modern LLMs tend to use 4bit inference, and Blackwell is significantly more optimized for 4 bit, we can see much less than 11 cents. Maybe a speed up of 5x if using 4bit and Blackwell vs H100 and 8 bit? So we're looking at potent…

[deleted]
Post reply on HN