Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

blog.kog.ai

Post reply on HN
Print view View original

41–50 of 108 posts

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#41

Post by gaeld »

CastFX wrote: post #35

Looks super promising! A couple of questions: For new open weights models, will you need to adapt model code and optimization for your inference engine by hand? It's true that BS=1 is king when it comes to agentic workflows, however these kinds of system serve multiple requests concurrently with dynamic batching. Do you think it will scale as well ? Any plans to release it open source? Congratz again for the release

Thanks a lot! Much appreciated.

To answer your questions:

- yes, we rewrite the whole model code (while keeping the same logic) in CUDA/HIP and assembly, in order to optimize by hand for each GPU type. It's quite tedious for sure, but I guess this is the price to pay to get this kind of results.

- the batching question is a great one. In agentic systems, there is probably a trade-off between sequential thinking/iterations vs parallel exploration of multiple solutions. Also, there could just be multiple independent tasks running in parallel, depending on the use case.

We plan to support a small amount of batching, but it quickly becomes a trade-off vs speed. Pick one for your use case, I guess.

Also to consider: because we answer requests much faster, we are also able to process lots of them without needing high batches - and scaling on multiple nodes is possible.

- open sourcing: maybe, maybe not. I'm still undecided on this. We are a small startup and I'm told that giving our IP away might be shooting ourselves in the feet. On the other side, I think it could be of great benefit to the community and for us... we'll see

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#42

Post by Hfuffzehn »

That's really nice of them.

That means Jensen can add another 30 times faster when comparing Rubin to Blackwell without having to actually do anything.

Hopefully that means he won't have any problem to make another 150 billion in profit in the next year.

Sorry for the sarcasm. Looks like interesting work.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#43

Post by Imustaskforhelp »

gaeld wrote: post #16
mungoman2 wrote: post #5

This looks very interesting. Possible to get those rates without exotic hardware. But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison. We need to see the comparison with this framework and useful models, which at present seems to m…

Great points. We strived to be fair as possible in the benchmark, but it's indeed not perfect. Taalas should have been added in the dedicated hardware section, even though they use 3-bit quantization when we are on FP16 (to be fair in both directions) and they burn the model directly on the card. Our tech preview is about the speed (hence the small dense model, it was easier to implement). The math checks out though…

Your playground/write-up is very interesting and I would be really interested when you can have something like Deepseek V4 Flash model (49B) running as you are suggesting.

I haven't read the article at the moment and I will try to read them hopefully but I wish to ask a question regarding, can this approach be done for say trillion or large parameter models as well or is there some wall which gets hit that makes it valuable for only smaller parameter model.

That being said, its still really incredible because in future, because these small models are really getting good for many use cases and speed becomes their bottleneck, with greater speeds at consumer hardware, I think its gonna be amazing work!

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#45

Post by yjftsjthsd-h »

Oras wrote: post #15
0-bad-sectors wrote: post #13

When I read "Standard GPUs" in the title I got excited for a second then I read the article itself..

what did you have in mind when you read "Standard GPUs"?

The GPU in my desktop. (A normal-ish decent gaming machine that runs LLMs and txt2img well enough.)

In contrast, not enterprise GPUs that cost as much as a car.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#46

Post by foobar10000 »

gaeld wrote: post #11
LoganDark wrote: post #6

I feel the comparison to Groq is unfair. They're running much larger models (orders of magnitude) and still reaching competitive speeds.

Fair point - this tech preview is about the speed (hence the small dense model, it was easier to implement). The math checks out though to allow support for large frontier MoE models at similar speeds. At batch size 1, GPT-OSS-120B has 5.1B active parameters - in FP8, it's in the same size ballpark than our 2B model in FP16 (5.1 GB vs 4GB). DeepSeek V4 Flash has 13B in mixed FP4/FP8. Check out the math at the end of…

[dead]

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#47

Post by gaeld »

Imustaskforhelp wrote: post #43
gaeld wrote: post #16

Earlier quoted context omitted.

Great points. We strived to be fair as possible in the benchmark, but it's indeed not perfect. Taalas should have been added in the dedicated hardware section, even though they use 3-bit quantization when we are on FP16 (to be fair in both directions) and they burn the model directly on the card. Our tech preview is about the speed (hence the small dense model, it was easier to implement). The math checks out though…

Your playground/write-up is very interesting and I would be really interested when you can have something like Deepseek V4 Flash model (49B) running as you are suggesting. I haven't read the article at the moment and I will try to read them hopefully but I wish to ask a question regarding, can this approach be done for say trillion or large parameter models as well or is there some wall which gets hit that makes it v…

Thanks for the comment and the question!

The last section of the article lays out the scaling laws that apply when porting this approach to another model. In a nutshell, DeepSeek V4 Pro with 49B active params is close to the upper bound.

Also worth noting that our results are currently for standard datacenter GPUs. On consumer hardware, though the same low-level optimization approach applies, the bandwidth limitations will cap the achievable speed.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#50

Post by bcjdjsndon »

Oras wrote: post #17
867-5309 wrote: post #7

> Standard GPUs > 8× NVIDIA H200

as not custom chips like Grog and Cerebras. Did you expect a single GPU chip to reach 3k tps?

> Did you expect a single GPU chip to reach 3k tps?

Did the article headline not say Standard GPU?

Post reply on HN