Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

blog.kog.ai

Post reply on HN
Print view View original

61–70 of 108 posts

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#61

Post by zozbot234 »

gaeld wrote: post #23

Follow-up reading the most technical and research people here: Monokernel deep dive (GPU Engineering): http://blog.kog.ai/building-a-single-kernel-latency-optimize... Delayed Tensor Parallelism (research): http://blog.kog.ai/delayed-tensor-parallelism-for-faster-tra... To try the speed on the playground: http://playground.kog.ai

It looks like DTP is a distinct architectural choice that would require training new models accordingly? This wouldn't be able to just run inference for existing models.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#62

Post by gaeld »

zozbot234 wrote: post #61
gaeld wrote: post #23

Follow-up reading the most technical and research people here: Monokernel deep dive (GPU Engineering): http://blog.kog.ai/building-a-single-kernel-latency-optimize... Delayed Tensor Parallelism (research): http://blog.kog.ai/delayed-tensor-parallelism-for-faster-tra... To try the speed on the playground: http://playground.kog.ai

It looks like DTP is a distinct architectural choice that would require training new models accordingly? This wouldn't be able to just run inference for existing models.

Totally, though DTP is not required for these kind of speeds. Standard TP works also.

DTP is something we built for our roadmap in order to get to extremely high speeds (like 10k+ tokens/s). When the budget is under 10 µs per layer, any little overhead matters.

For 1k to 5k tokens/s, regular TP still works because we are able to optimize the inter-GPU all-reduce collectives at under 3 µs, which allows to continue streaming model weights in shared memory, registers and caches while GPUs exchange data.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#63

Post by gaeld »

bcjdjsndon wrote: post #54
gaeld wrote: post #19

Earlier quoted context omitted.

thanks! we explain how it scales to larger models in the last section the OP blog post

Shame you stopped short of actually benchmarking that scale though, eh?

will do - we are a small team and it takes time to implement and optimize a new model, whatever the size.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#65

Post by gaeld »

frankensteins wrote: post #48

I have a naive question here - first, the token speed is very impressive. but why this is the highlight? I would prefer the actual performance.

Token generation speed matters for sequential agentic workflows, like software engineering / vibe coding, where a lot of reasoning tokens, code generation, refactoring, testing, etc. happen in a loop before an actual outcome is served to the user.

About model performance, we plan to support the latest frontier models (this tech preview is about the speed of the engine)

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#66

Post by gaeld »

deflator wrote: post #51
gaeld wrote: post #21

Earlier quoted context omitted.

I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed.

What a lot of use on here are salivating for is the ability to run these on prosumer hardware at home. So we tend to jump to the conclusion that "standard" means "consumer-grade" because that's what we want to see. Still, very cool work!

thank you deflator, I understand this now! much appreciated

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#67

Post by nightski »

gaeld wrote: post #21
Oras wrote: post #15

Earlier quoted context omitted.

what did you have in mind when you read "Standard GPUs"?

I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed.

How would you classify a datacenter GPU as standard/non-standard? That doesn't seem to be a meaningful distinction. It's click bait.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#68

Post by zozbot234 »

Imustaskforhelp wrote: post #43
gaeld wrote: post #16

Earlier quoted context omitted.

Great points. We strived to be fair as possible in the benchmark, but it's indeed not perfect. Taalas should have been added in the dedicated hardware section, even though they use 3-bit quantization when we are on FP16 (to be fair in both directions) and they burn the model directly on the card. Our tech preview is about the speed (hence the small dense model, it was easier to implement). The math checks out though…

Your playground/write-up is very interesting and I would be really interested when you can have something like Deepseek V4 Flash model (49B) running as you are suggesting. I haven't read the article at the moment and I will try to read them hopefully but I wish to ask a question regarding, can this approach be done for say trillion or large parameter models as well or is there some wall which gets hit that makes it v…

Consumer inference scenarios tend to be highly bespoke so it's difficult to apply a monokernel approach based on deep manual optimization. I suppose this could become applicable to rare scenarios where both the model and the hardware are fixed and self-contained, e.g. I'm running Apple's AI model on the latest Apple Silicon hardware. Then this becomes a viable approach even for 'consumer' use.

The authors' approach also encompasses multi-node approaches that won't apply easily to consumer inference since consumer GPUs have very low-performance interconnects, hence why layer parallelism is usually favored. (But that doesn't work very well with the monokernel approach, since it involves running distinct logic on each separate GPU. It also doesn't speed up single inference, though you can get that throughput back by pipelining small minibatches.)

Post reply on HN