Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

blog.kog.ai

Post reply on HN
Print view View original

81–90 of 108 posts

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#81

Post by gaeld »

joefourier wrote: post #79

Do you think the work will still apply to speculative/alternative decoding methods like MTP and block diffusion, which are making batch=1 decoding less memory bound? Kernel launch overhead and memory transfer become less and less significant as a % of time when computing multiple tokens at once.

Why not, it's one way to look at it! Although I have yet to see other work with speculative decoding higher than ~1,000 tokens/s., because the other bottlenecks start to matter at that point, and they need to be solved to go further.

Our view is that MTP / speculative decoding could help getting a X multiplier (X = 2 to 6) on the tokens per second speed we currently achieve.

We are a bit greedy, we want to stack optimizations on top of each other to get the maximum speed possible.

It involves additional compute to verify the predicted tokens during the forward pass (it's like a small batch), which should be totally doable for dense models, and will be more tricky for MoEs because it could mean activating more experts and thus more active parameters.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#83

Post by blindr »

For me it's 3.4k tok/s of pure nonsense, the model is bad, you tell it it's wrong, it acknowledge it's wrong and repeats the same nonsense. It reminds me my nephew though. Ask it something like: "I want to play the guitar on the surface of the Moon. What speakers do you suggest." and then "But Moon has no atmosphere, how the sound will travel?".

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#84

Post by wmf »

zozbot234 wrote: post #68
Imustaskforhelp wrote: post #43

Earlier quoted context omitted.

Your playground/write-up is very interesting and I would be really interested when you can have something like Deepseek V4 Flash model (49B) running as you are suggesting. I haven't read the article at the moment and I will try to read them hopefully but I wish to ask a question regarding, can this approach be done for say trillion or large parameter models as well or is there some wall which gets hit that makes it v…

Consumer inference scenarios tend to be highly bespoke so it's difficult to apply a monokernel approach based on deep manual optimization. I suppose this could become applicable to rare scenarios where both the model and the hardware are fixed and self-contained, e.g. I'm running Apple's AI model on the latest Apple Silicon hardware. Then this becomes a viable approach even for 'consumer' use. The authors' approach a…

scenarios where both the model and the hardware are fixed and self-contained

That's basically antirez's DS4 and it works pretty well because there are few leading models and few hardware platforms (Apple, GB10, Strix Halo) that are worth using.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#85

Post by gaeld »

blindr wrote: post #83

For me it's 3.4k tok/s of pure nonsense, the model is bad, you tell it it's wrong, it acknowledge it's wrong and repeats the same nonsense. It reminds me my nephew though. Ask it something like: "I want to play the guitar on the surface of the Moon. What speakers do you suggest." and then "But Moon has no atmosphere, how the sound will travel?".

Note that this coding model is trained on programming use cases, and is also not tuned for multi-turn chat.

You can ask it to implement an algorithm; we provide suggested prompts you can test.

Also, this tech preview is really about the speed of the inference engine (not the model itself) so I'm glad you got 3.4k tok/s!

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#86

Post by gaeld »

dr_kiszonka wrote: post #80
gaeld wrote: post #32

Earlier quoted context omitted.

Great points, let me clarify: - model size: 2B is just for this preview (it was faster to implement), our article explains how we expect to support large frontier MoE at 1,000 to 5,000 tokens/s - reaching 500 tok/s, or even up to ~1,000 tok/s, on a consumer GPU card is possible with existing inference engines like vLLM. But there is a ceiling. The hard part comes we you try to be faster than that: these frameworks wo…

Thank you for explaining. Do you think there are still opportunities for stack optimizations to meaningfully speed up inference on single consumer-grade GPUs?

I'm sure there are, and I really hope we can work on consumer-grade GPUs at some point.

It should be possible to apply the same methodology (digging deep into the hardware details to understand all its little characteristics, and rethinking the inference stack around that).

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#88

Post by freediddy »

Who cares about token speed? What is the quality of the results like? I don't know why people are so fixated on token speed, since no one cares how quickly it can spew garbage. Most reasonable people are happier waiting a bit more for accurate results.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#89

Post by gaeld »

DeathArrow wrote: post #82

>This preview runs a 2B model I guess with 1B or 500M model inference would be even faster?

In theory yes, although not in a linearly proportional way, because in practice our memory streaming is not yet perfect. There are still some fixed costs that we did not fully optimize (for now).

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#90

Post by esafak »

freediddy wrote: post #88

Who cares about token speed? What is the quality of the results like? I don't know why people are so fixated on token speed, since no one cares how quickly it can spew garbage. Most reasonable people are happier waiting a bit more for accurate results.

It matters on consumer hardware since barely any model runs at reasonable speed.
Post reply on HN