When I read "Standard GPUs" in the title I got excited for a second then I read the article itself..
what did you have in mind when you read "Standard GPUs"?
I guessed you thought about consumer GPUs.
We are about standard datacenter GPUs indeed.
Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
21–30 of 108 posts
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#22NVIDIA H200 Is not a standard GPU. 8 of them in a box with a cpu and ram costs close to the same as a house. I am 100% all about using local models instead of sending someone else all my data and paying for the privilege of doing so, this article is misleading. I can get a 27b model to kick out 40 tok/s on 16 gb vram. This is the area ripe for development. If you can’t connect a monitor, it isn’t a standard GPU, at l…
I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed.
Sorry for the confusion
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#23Follow-up reading the most technical and research people here:
Monokernel deep dive (GPU Engineering): http://blog.kog.ai/building-a-single-kernel-latency-optimize...
Delayed Tensor Parallelism (research): http://blog.kog.ai/delayed-tensor-parallelism-for-faster-tra...
To try the speed on the playground: http://playground.kog.ai
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#24When I read "Standard GPUs" in the title I got excited for a second then I read the article itself..
Yeah, it should have been "Datacenter GPUs" or "Nvidia and AMD GPUs".
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#25> Standard GPUs > 8× NVIDIA H200
as not custom chips like Grog and Cerebras. Did you expect a single GPU chip to reach 3k tps?
I think many would assume "not enterprise" or "not datacenter grade" when someone says "Standard GPUs", but maybe that specific phrase have a specific meaning I'm not familiar with.
Edit: I just tried a 4B model on a RTX Pro 6000, getting ~500 tok/s with llama.cpp not even trying to optimize or change anything, just default settings. I'm sure with vLLM it'd be a lot faster already, still before manually tuning configs. I wouldn't call that card "Standard GPU" either FWIW, but it makes the claimed performance numbers feel not as exciting, especially given the hardware they were using.
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#26Earlier quoted context omitted.
as not custom chips like Grog and Cerebras. Did you expect a single GPU chip to reach 3k tps?
I think many would assume "not enterprise" or "not datacenter grade" when someone says "Standard GPUs", but maybe that specific phrase have a specific meaning I'm not familiar with. Edit: I just tried a 4B model on a RTX Pro 6000, getting ~500 tok/s with llama.cpp not even trying to optimize or change anything, just default settings. I'm sure with vLLM it'd be a lot faster already, still before manually tuning config…
I expected a 4090, maybe 2.
I did not expect 8xH200 for a 2B model.
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#27NVIDIA H200 Is not a standard GPU. 8 of them in a box with a cpu and ram costs close to the same as a house. I am 100% all about using local models instead of sending someone else all my data and paying for the privilege of doing so, this article is misleading. I can get a 27b model to kick out 40 tok/s on 16 gb vram. This is the area ripe for development. If you can’t connect a monitor, it isn’t a standard GPU, at l…
I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed. Sorry for the confusion
Do you think maybe changing your articles title from "Real-time LLM Inference on Standard GPUs" to "Real-time LLM Inference on Standard Datacenter GPUs" might make sense here? Given more people seem confused by the title than not, and you could clear this up relatively easily, at least on your website although might be late to fix the HN title.
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#28I can think of real time video, shader generation, real time worldbuilding type problems could require such a high token throughput. For instant code generatio, 400-500 tok/s should be sufficient, though most frontier models give us closer to 70 tok/s.
That sounds a little bit like the 64kb memory is enough, then someone invented electron ;P
But joke aside, I think we don't even know yet what is possible if you hit very fast very high token / second numbers if your whole ecosystem behind it can handle it.
You could literaly implement the same solution 100x and benchmark all of them and get only the best result.
You could build and architecture a whole stack in parallel.
You could do massive thinking token / chain of thought.
You could let the LLM analyse everything around you while you type. Like it could tell you that this might create a bug in a different file and why.
We could start doing some type of monte-carlo search with this.
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#29NVIDIA H200 Is not a standard GPU. 8 of them in a box with a cpu and ram costs close to the same as a house. I am 100% all about using local models instead of sending someone else all my data and paying for the privilege of doing so, this article is misleading. I can get a 27b model to kick out 40 tok/s on 16 gb vram. This is the area ripe for development. If you can’t connect a monitor, it isn’t a standard GPU, at l…
I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed. Sorry for the confusion
Oh, it isn't confusing, it is misleading. A standard GPU lets you connect a monitor. A datacenter GPU lets you do headless math.
Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
#30Earlier quoted context omitted.
I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed. Sorry for the confusion
Do you think maybe changing your articles title from "Real-time LLM Inference on Standard GPUs" to "Real-time LLM Inference on Standard Datacenter GPUs" might make sense here? Given more people seem confused by the title than not, and you could clear this up relatively easily, at least on your website although might be late to fix the HN title.
YES - I just updated the title of our article according to your suggestion.