Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

blog.vllm.ai

Post reply on HN
Print view View original

31–40 of 61 posts

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#31

Post by ffsm8 »

supermatt wrote: post #28
t0mas88 wrote: post #27

Earlier quoted context omitted.

That's excluding tax, net prices around 0.20-0.30 EUR / Kwh we common.

I updated my comment to include my personal delivered rate including VAT - also note that businesses (like a data center) don't pay the VAT and have substantially reduced delivery fees at high voltage

Then you're living in one of the cheapest areas for electricity prices in Europe, the opposite of what you said.

https://ec.europa.eu/eurostat/statistics-explained/index.php...

Scroll a little down and you see a breakdown by country

E.g.

https://ec.europa.eu/eurostat/statistics-explained/index.php...

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#33

Post by menaerus »

dust42 wrote: post #16

If I followed the links correctly this benchmark was made on a 16xH200. At current prices I'd assume that is a system price of around $750,000. The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages. Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated wi…

How did you arrive to $10,000 electricity costs figure?

8xH200 enclosed in DGX H200 system power draw is ~14kW in its peak (CTS) configuration/utilization. Over one year, and assuming maximum utilization, this is 123,480 kWh per single DGX H200 unit. We need 2x such units for 16xH200 system configuration under subject so it's 246,960 kWh/year. This is ~$25,000 at 10cts per kWh and ~$74,000 at 30cts per kWh. At ~1,110,000 1M batches this gives us: (1) ~$0.02 - $0.07 per 1M of energy cost and (2) ~$0.25 per 1M assuming the same HW depreciation rate. In total, this is ~$0.3 per 1M tokens.

Seems sustainable?

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#34

Post by bildung »

sofixa wrote: post #23
androiddrew wrote: post #4

Now all we need is better support for AMD gpus, both CDNA and RDNA types

You can run vLLM with AMD GPUs supported by ROCm: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/infer... However from experience with an AMD Strix Halo, a couple of caveats: it's drastically slower than Ollama (tested over a few weeks, always using the official AMD vLLM nightly releases), and not all GPUs were supported for all models (but that has been fixed).

vLLM ususally only plays out its strength when serving multiple users in parallel, in contrast to llama.cpp (Ollama is a wrapper around llama.cpp).

If you want more performance, you could try running llama.cpp directly or use the prebuilt lemonade nightlies.

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#35

Post by supermatt »

ffsm8 wrote: post #31
supermatt wrote: post #28

Earlier quoted context omitted.

I updated my comment to include my personal delivered rate including VAT - also note that businesses (like a data center) don't pay the VAT and have substantially reduced delivery fees at high voltage

Then you're living in one of the cheapest areas for electricity prices in Europe, the opposite of what you said. https://ec.europa.eu/eurostat/statistics-explained/index.php... Scroll a little down and you see a breakdown by country E.g. https://ec.europa.eu/eurostat/statistics-explained/index.php...

I am in Lithuania, which has one of the highest wholesale energy prices in Europe (as per nord pool): https://data.nordpoolgroup.com/auction/day-ahead/prices?deli...

That it is not translating into a higher cost to the consumer (as evidenced on your link) is likely indicative of other costs being incurred by the “average” consumer in those countries with a higher domestic rate - like massive markup from users being tied into inflated contracts due to the 2022 shock where rates across Europe were more than double what they are now.

Also, these are residential prices - business prices are usually much lower (wholesale discounts, subsidies, no VAT, lower delivery charges).

As per my response to the initial comment - there is no way a datacentre in Europe is paying 30c/kWh

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#36

Post by Palmik »

dust42 wrote: post #16

If I followed the links correctly this benchmark was made on a 16xH200. At current prices I'd assume that is a system price of around $750,000. The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages. Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated wi…

APIs are usually very profitable. As for subscriptions, it would depend on how many tokens average subscriber uses per month. Do we have some source of info on this?

Some notes:

- # Input tokens & # output tokens per request matters a lot.

- KV Cache hit rate matters a lot.

- vLLM is not the necessarily most efficient engine.

- You are looking at API cost for DeepSeek V3.2, which is much cheaper than DeepSeek R1 / V3 / V3.1. DeepSeek V3.2 is different architecture (sparse attention) that is much more efficient. DeepSeek V3 cheapest option (fp8) tends to be ~$1/mil output tokens while R1 tends to be ~$2.5/mil (note that for example Together AI charges whopping $7/mil output tokens for R1!)

As for the cost: You can also get H200s for ~ $1.6/hr and H100s for ~ $1.2/hr. That somewhat simplifies the calculations :)

Ignoring the caveats and assuming H200s, with their setup you will:

- Process 403200000 input tokens.

- Generate 126720000 output tokens.

- Spend $25.6.

- On Together with DS R1 it would cost you $3 * 403.2 + $7 * 126.7 = ~$2096. Together does not even offer discount for KV cache hits (what a joke :)).

- On NovitaAI with DS R1 it would cost you $0.7 * 403.2 + $2.5 * 126.7 = ~$600 (with perfect cache hit rate, which gives 50% discount on input tokens here, it would be ~$458).

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#37

Post by menaerus »

supermatt wrote: post #35
ffsm8 wrote: post #31

Earlier quoted context omitted.

Then you're living in one of the cheapest areas for electricity prices in Europe, the opposite of what you said. https://ec.europa.eu/eurostat/statistics-explained/index.php... Scroll a little down and you see a breakdown by country E.g. https://ec.europa.eu/eurostat/statistics-explained/index.php...

I am in Lithuania, which has one of the highest wholesale energy prices in Europe (as per nord pool): https://data.nordpoolgroup.com/auction/day-ahead/prices?deli... That it is not translating into a higher cost to the consumer (as evidenced on your link) is likely indicative of other costs being incurred by the “average” consumer in those countries with a higher domestic rate - like massive markup from users being t…

Some countries also employ progressive electricity pricing such that higher energy consumption leads to elevated kWh rates incentivizing conservation. This is also not visible in the stats above. I also think that business kWh rates are actually higher than for the households in some instances.

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#39

Post by Palmik »

I wish there were more open benchmarks comparing different setups and different engines. There are so many knobs to tune (TP / DP / PP / PD / spec. decoding / etc.) and while the optimal setup will be highly dependent on the model, the environment and the traffic, it's likely some useful conclusions could be drawn.

It almost feels like in the past year there is some unwritten agreement between the 3 main open-source engines (vLLM, sglang, TRT-LLM) to not compare to each other directly :) They used to publish benchmarks comparing against each other quite regularly.

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#40

Post by bonoboTP »

t0mas88 wrote: post #27
supermatt wrote: post #25

Earlier quoted context omitted.

> more like 30cts like it is in Europe Nope - i live in one of the most expensive areas, and even the residential price has averaged 18c/kWh delivered including taxes. Businesses get a lower basic rate and also don't pay the VAT, so it works out around 13c/kWh for them. https://data.nordpoolgroup.com/auction/day-ahead/prices?deli...

That's excluding tax, net prices around 0.20-0.30 EUR / Kwh we common.

Net is excluding tax, you mean gross.
Post reply on HN