Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

blog.kog.ai

Post reply on HN
Print view View original

11–20 of 108 posts

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#11

Post by gaeld »

LoganDark wrote: post #6

I feel the comparison to Groq is unfair. They're running much larger models (orders of magnitude) and still reaching competitive speeds.

Fair point - this tech preview is about the speed (hence the small dense model, it was easier to implement).

The math checks out though to allow support for large frontier MoE models at similar speeds.

At batch size 1, GPT-OSS-120B has 5.1B active parameters - in FP8, it's in the same size ballpark than our 2B model in FP16 (5.1 GB vs 4GB).

DeepSeek V4 Flash has 13B in mixed FP4/FP8.

Check out the math at the end of our blog post: https://blog.kog.ai/real-time-llm-inference-on-standard-gpus...

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#12

Post by kirtivr »

mungoman2 wrote: post #5

This looks very interesting. Possible to get those rates without exotic hardware. But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison. We need to see the comparison with this framework and useful models, which at present seems to m…

They got 1K tok/s with Deepseek v4 Pro. That's kinda cool..

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#16

Post by gaeld »

mungoman2 wrote: post #5

This looks very interesting. Possible to get those rates without exotic hardware. But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison. We need to see the comparison with this framework and useful models, which at present seems to m…

Great points.

We strived to be fair as possible in the benchmark, but it's indeed not perfect. Taalas should have been added in the dedicated hardware section, even though they use 3-bit quantization when we are on FP16 (to be fair in both directions) and they burn the model directly on the card.

Our tech preview is about the speed (hence the small dense model, it was easier to implement).

The math checks out though to allow support for large frontier MoE models at similar speeds: - At batch size 1, GPT-OSS-120B has 5.1B active parameters - in FP8, it's in the same size ballpark than our 2B model in FP16 (5.1 GB vs 4GB). - DeepSeek V4 Flash has 13B in mixed FP4/FP8, so let's say ballpark around 3x bigger than 4GB - so in theory we could reach >1,000 tok/s on it with MI300X/H200 and up to 4k on next generation GPUs.

Check out the math at the end of our blog post:

https://blog.kog.ai/real-time-llm-inference-on-standard-gpus...

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#18

Post by gaeld »

kirtivr wrote: post #12
mungoman2 wrote: post #5

This looks very interesting. Possible to get those rates without exotic hardware. But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison. We need to see the comparison with this framework and useful models, which at present seems to m…

They got 1K tok/s with Deepseek v4 Pro. That's kinda cool..

Thanks. To be fair, this number is what we expect to get once we port DeepSeek V4 in our engine on the upcoming generation of GPUs!

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#19

Post by gaeld »

ilaksh wrote: post #4

Could be amazing, but it's hard to judge if it will really work with say a 27 B model or larger. We can already get pretty good speed with a 2B model.

thanks! we explain how it scales to larger models in the last section the OP blog post

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#20

Post by irishcoffee »

NVIDIA H200 Is not a standard GPU. 8 of them in a box with a cpu and ram costs close to the same as a house.

I am 100% all about using local models instead of sending someone else all my data and paying for the privilege of doing so, this article is misleading.

I can get a 27b model to kick out 40 tok/s on 16 gb vram. This is the area ripe for development.

If you can’t connect a monitor, it isn’t a standard GPU, at least not in the way people have spoken about GPUs until a few years ago.

Post reply on HN