Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
1–10 of 108 posts
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#2[deleted]
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#3[dead]
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#4Could be amazing, but it's hard to judge if it will really work with say a 27 B model or larger. We can already get pretty good speed with a 2B model.
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#5This looks very interesting. Possible to get those rates without exotic hardware.
But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison.
We need to see the comparison with this framework and useful models, which at present seems to mean ~30 B.
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#6I feel the comparison to Groq is unfair. They're running much larger models (orders of magnitude) and still reaching competitive speeds.
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#7> Standard GPUs
> 8× NVIDIA H200
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#8I can think of real time video, shader generation, real time worldbuilding type problems could require such a high token throughput.
For instant code generatio, 400-500 tok/s should be sufficient, though most frontier models give us closer to 70 tok/s.
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#9> Standard GPUs > 8× NVIDIA H200
Everyone beholden to a data center or subject to the installation on the corner of your property of course. Keep up with the times... /s
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#10This looks very interesting. Possible to get those rates without exotic hardware. But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison. We need to see the comparison with this framework and useful models, which at present seems to m…
likely the small model makes whatever fuzzer they designed to poke the gpus much faster optimizations.
they seem to think it scales up because theyre shortening the stack.