Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

blog.vllm.ai

Post reply on HN
Print view View original

51–60 of 61 posts

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#51

Post by ffsm8 »

supermatt wrote: post #35
ffsm8 wrote: post #31

Earlier quoted context omitted.

Then you're living in one of the cheapest areas for electricity prices in Europe, the opposite of what you said. https://ec.europa.eu/eurostat/statistics-explained/index.php... Scroll a little down and you see a breakdown by country E.g. https://ec.europa.eu/eurostat/statistics-explained/index.php...

I am in Lithuania, which has one of the highest wholesale energy prices in Europe (as per nord pool): https://data.nordpoolgroup.com/auction/day-ahead/prices?deli... That it is not translating into a higher cost to the consumer (as evidenced on your link) is likely indicative of other costs being incurred by the “average” consumer in those countries with a higher domestic rate - like massive markup from users being t…

Business prices should be figure 6 in my link, while the difference is a lot smaller, Lithuania is definitely one of the cheaper countries, beating the EU average slightly.

> As per my response to the initial comment - there is no way a datacentre in Europe is paying 30c/kWh

Hetzner prices it at 33c/wh as of last year I believe, previously it was 40c (after the pipeline was destroyed)

But Germany is pretty much in the 3 most expensive countries wrt electricity cost in the EU - both for consumers and commercial pricing

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#52

Post by supermatt »

ffsm8 wrote: post #51
supermatt wrote: post #35

Earlier quoted context omitted.

I am in Lithuania, which has one of the highest wholesale energy prices in Europe (as per nord pool): https://data.nordpoolgroup.com/auction/day-ahead/prices?deli... That it is not translating into a higher cost to the consumer (as evidenced on your link) is likely indicative of other costs being incurred by the “average” consumer in those countries with a higher domestic rate - like massive markup from users being t…

Business prices should be figure 6 in my link, while the difference is a lot smaller, Lithuania is definitely one of the cheaper countries, beating the EU average slightly. > As per my response to the initial comment - there is no way a datacentre in Europe is paying 30c/kWh Hetzner prices it at 33c/wh as of last year I believe, previously it was 40c (after the pipeline was destroyed) But Germany is pretty much in th…

> Lithuania is definitely one of the cheaper countries

And yet has one of the highest wholesale rates...

> Hetzner prices it at...

Hertzner are reselling. They make a profit on energy resale. Their rate also includes a substantial buffer on the actual rate to account for volatility. Their rate is most likely less than half of what they are passing on for colo.

For reference, last year German industrial energy prices were around 10c/kWh INCLUDING taxes and network fees - and the government are looking to subsidize that further to target 5c/kWh: https://www.gleisslutz.com/en/know-how/germany-cuts-costs-el...

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#53

Post by ffsm8 »

supermatt wrote: post #52
ffsm8 wrote: post #51

Earlier quoted context omitted.

Business prices should be figure 6 in my link, while the difference is a lot smaller, Lithuania is definitely one of the cheaper countries, beating the EU average slightly. > As per my response to the initial comment - there is no way a datacentre in Europe is paying 30c/kWh Hetzner prices it at 33c/wh as of last year I believe, previously it was 40c (after the pipeline was destroyed) But Germany is pretty much in th…

> Lithuania is definitely one of the cheaper countries And yet has one of the highest wholesale rates... > Hetzner prices it at... Hertzner are reselling. They make a profit on energy resale. Their rate also includes a substantial buffer on the actual rate to account for volatility. Their rate is most likely less than half of what they are passing on for colo. For reference, last year German industrial energy prices…

You're talking about select industries which are being supported via subventions, data centers are not included. If you pay attention to the wording in your cited article, they've said so as well.

And hetzner does not have a large upsell for their energy prices, they're pretty much passing in the price as-is according to their own statements (from the large increase to 40c)

Almost all commercial applications need to pay the quoted prices around what's shown in figure 6

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#54

Post by rbanffy »

mzl wrote: post #32
rbanffy wrote: post #22

Very impressive numbers - I'd expect 2K tok/s on Cerebras hardware, not H200's.

I don't think it would be economically viable to serve the full DeepSeek models on Cerebras hardware.

I'm a huge fan of their hardware - we've been promimsed wafer-scale integration since the 1980s and they delivered it. It'd be a shame if their tech ended up a dead-end.

On the bright side, they haven't started exploring stacking chips on top of their wafers to increase local memory, and every process change will bring increased bandwidth in and out of their "pizza". I really wish they succeed.

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#55

Post by mycelia »

aurareturn wrote: post #48
mycelia wrote: post #26

Hey HN! I’m Seiji Eicher from Anyscale, one of the authors of this post :) Feel free to ask questions here.

Are you using 16bit for inference? How many tokens/second if you use 8bit? Given that SOTA models now use 4bit inference, can you do an estimation for 4bit + Blackwell?

Hi! This benchmarking was done w/ DeepSeek-V3's published FP8 weights. And Blackwell performance is still being optimized. SGLang hit 14k/s/B200 though, pretty cool writeup here: https://lmsys.org/blog/2025-09-25-gb200-part-2/

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#56

Post by mycelia »

Palmik wrote: post #38
mycelia wrote: post #26

Hey HN! I’m Seiji Eicher from Anyscale, one of the authors of this post :) Feel free to ask questions here.

Great work! What optimizations are you most excited about for 2026?

Lot of cool stuff coming up! As a Ray developer, I focus more on the orchestration layer, so I'm excited about things like Elastic Expert Parallelism, posttraining enhancements like colocated trainer/engines, and deploying DSV4 (rumors are the architecture will be complex). vLLM roadmap is here for reference: http://roadmap.vllm.ai/

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#57

Post by mycelia »

menaerus wrote: post #29
mycelia wrote: post #26

Hey HN! I’m Seiji Eicher from Anyscale, one of the authors of this post :) Feel free to ask questions here.

Do you use agentic AI yet for this type of optimization work or no?

For my work personally, agentic AI usage is pretty standard SWE fare (Cursor/CC). Even within the engine, optimizations are often centered around things like increasing communication/compute overlap (this is called Dual-Batch Overlap in vLLM).

Probably there are more interesting/easily verifiable agent loops you could try for kernel optimizations. At this point, the best are still written by hand, though. Ex: DeepEP kernels https://github.com/deepseek-ai/DeepEP

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#58

Post by LTL_FTC »

rbanffy wrote: post #54
mzl wrote: post #32

Earlier quoted context omitted.

I don't think it would be economically viable to serve the full DeepSeek models on Cerebras hardware.

I'm a huge fan of their hardware - we've been promimsed wafer-scale integration since the 1980s and they delivered it. It'd be a shame if their tech ended up a dead-end. On the bright side, they haven't started exploring stacking chips on top of their wafers to increase local memory, and every process change will bring increased bandwidth in and out of their "pizza". I really wish they succeed.

Well it looks like as you were typing your comment, a press release was going out announcing OpenAi’s $10B investment in Cerebras.

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#59

Post by mzl »

rbanffy wrote: post #54
mzl wrote: post #32

Earlier quoted context omitted.

I don't think it would be economically viable to serve the full DeepSeek models on Cerebras hardware.

I'm a huge fan of their hardware - we've been promimsed wafer-scale integration since the 1980s and they delivered it. It'd be a shame if their tech ended up a dead-end. On the bright side, they haven't started exploring stacking chips on top of their wafers to increase local memory, and every process change will bring increased bandwidth in and out of their "pizza". I really wish they succeed.

Oh, I'm also a fan. It is really cool to see what they've done. However, in the current systems they have available, they would (as far as I've understood it) just need way to many racks to be able to serve the full Deepseek model for it to have any kind of economics. The main limiting factor is the amount of sram available per wafer.

Re: vLLM large scale serving: DeepSeek 2.2k tok/s/h200 with wide-ep

#60

Post by supermatt »

ffsm8 wrote: post #53
supermatt wrote: post #52

Earlier quoted context omitted.

> Lithuania is definitely one of the cheaper countries And yet has one of the highest wholesale rates... > Hetzner prices it at... Hertzner are reselling. They make a profit on energy resale. Their rate also includes a substantial buffer on the actual rate to account for volatility. Their rate is most likely less than half of what they are passing on for colo. For reference, last year German industrial energy prices…

You're talking about select industries which are being supported via subventions, data centers are not included. If you pay attention to the wording in your cited article, they've said so as well. And hetzner does not have a large upsell for their energy prices, they're pretty much passing in the price as-is according to their own statements (from the large increase to 40c) Almost all commercial applications need to…

Ok, it seems I am mistaken that this subsidy applies to datacenter (apparently there is ongoing discussions to include them for this reason).

That said - I 100% don't believe that hertzner are simply passing on the price for their colo clients. Where did you read that they are not making a profit off electricity resale?

Here is another link discussing industrial energy prices WITHOUT reductions: https://www.smard.de/page/en/topic-article/213922/216044

So less than 17c/kWh in 2024, and likely another 2c when adjusted for current wholesale prices and network fees.

Post reply on HN