51–60 of 108 posts
Earlier quoted context omitted.
what did you have in mind when you read "Standard GPUs"?
I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed.
Earlier quoted context omitted.
I expected a 4090, maybe 2. I did not expect 8xH200 for a 2B model.
Great points, let me clarify: - model size: 2B is just for this preview (it was faster to implement), our article explains how we expect to support large frontier MoE at 1,000 to 5,000 tokens/s - reaching 500 tok/s, or even up to ~1,000 tok/s, on a consumer GPU card is possible with existing inference engines like vLLM. But there is a ceiling. The hard part comes we you try to be faster than that: these frameworks wo…
Could be amazing, but it's hard to judge if it will really work with say a 27 B model or larger. We can already get pretty good speed with a 2B model.
thanks! we explain how it scales to larger models in the last section the OP blog post
each time getting 3300+ tps.
The demo is very impressive!
disclaimer: I've known the founder for a while, as legitimate as it gets in deep tech, real years of research and engineering behind this, not vaporware
Feels like a preview of the future
> Standard GPUs > 8× NVIDIA H200
I have been lamenting for a while that the memory-bandwidth tps relationship was pretty much working for small models on consumer cards, but not at all on datacenter hardware.
It's great to see that with proper care on the inference engine implementation the relationship can be restored.