cool.. so now approximations of copies of copies can be approximated and copied faster?
Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
91–100 of 108 posts
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#92Who cares about token speed? What is the quality of the results like? I don't know why people are so fixated on token speed, since no one cares how quickly it can spew garbage. Most reasonable people are happier waiting a bit more for accurate results.
It matters on consumer hardware since barely any model runs at reasonable speed.
It also matters for thinking models and for agentic workflows, especially in software engineering, where a lot of tokens need to be output in iterative loops before the user sees any result.
This is our main use case.
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#93This looks very interesting. Possible to get those rates without exotic hardware. But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison. We need to see the comparison with this framework and useful models, which at present seems to m…
They got 1K tok/s with Deepseek v4 Pro. That's kinda cool..
No they didn't, they predict they'll get that much. Also worth noting the prediction assumes running at MXFP4/FP8 quantization.
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#94Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#95Earlier quoted context omitted.
Shame you stopped short of actually benchmarking that scale though, eh?
will do - we are a small team and it takes time to implement and optimize a new model, whatever the size.
You don't even need to train the model just to see if you can infer it at the claimed speed
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#96Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#97a "standard" GPU would be an nvidia geforce RTX 5090
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#98Earlier quoted context omitted.
will do - we are a small team and it takes time to implement and optimize a new model, whatever the size.
You don't even need to train the model just to see if you can infer it at the claimed speed
True, and for third-party models we'll just re-use their public open weights.
There is a time-consuming part, though, that is performed manually by our (human) team: implement the logic of the model in C++ and assembly code in a super-optimized way, co-designed for each specific hardware card.
This can take months.
We hope to accelerate the process with AI agents, but we're not there yet.
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#99Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#100I know all is relative but when I think of
> 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200
I still find it mind boggling. That's a lot of compute power and still considered "low end" for the purpose it serves.