Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Surpassing vLLM with a Generated Inference Stack

infinity.inc

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Surpassing vLLM with a Generated Inference Stack

#11

Post by rfw300 »

lukebechtel wrote: post #4
rfw300 wrote: post #3

OK... we need way more information than this to validate this claim! I can run Qwen-8B at 1 billion tokens per second if you don't check the model's output quality. No information is given about the source code, correctness, batching, benchmark results, quantization, etc. etc. etc.

We validate with MMLU and Hellaswag presently, and are getting this independently verified by a 3rd party. We have considered open-sourcing some of our optimized inference libraries in the future, but have not yet come to a decision on this. Also if you need a rough intuition as to why this is possible: it's because this entire inference stack was built for exactly one model, and thus we can really tune the entire fr…

I've no problem with the intuition. But I would hope for a lot more focus in the marketing materials on proving the (statistical) correctness of the implementation. 15% better inference speed is not worth it to use a completely unknown inference engine not tested across a wide range of generation scenarios.

Re: Surpassing vLLM with a Generated Inference Stack

#12

Post by lukebechtel »

rfw300 wrote: post #11
lukebechtel wrote: post #4

Earlier quoted context omitted.

We validate with MMLU and Hellaswag presently, and are getting this independently verified by a 3rd party. We have considered open-sourcing some of our optimized inference libraries in the future, but have not yet come to a decision on this. Also if you need a rough intuition as to why this is possible: it's because this entire inference stack was built for exactly one model, and thus we can really tune the entire fr…

I've no problem with the intuition. But I would hope for a lot more focus in the marketing materials on proving the (statistical) correctness of the implementation. 15% better inference speed is not worth it to use a completely unknown inference engine not tested across a wide range of generation scenarios.

This is a fair critique! We plan to use our system to generate many more inference libraries of this nature, and I'll make it a point to release better, broader correctness measures when we do so.

Re: Surpassing vLLM with a Generated Inference Stack

#13

Post by 2001zhaozhao »

Every example like this makes it obvious that you can now use ML-like optimization approaches on well-specified, very-well-tested software problems with a clear optimization goal. Keep if it improves the objective while maintaining correctness, discard if it doesn't. AI-descent strikes again.

Maybe I should learn more about ML to have a better instinct on optimization methods in general, so I can actually build AI optimizers like these.

Re: Surpassing vLLM with a Generated Inference Stack

#15

Post by jeeeb »

ntonozzi wrote: post #2

Why do they need to run benchmarks to confirm performance? Can't they run an example prompt and verify they get the exact same output token probabilities for all prompts? The fact that they are not doing this makes me suspicious that they are in fact not doing the exact same thing as vLLM. It is also a bit weird that they are not incorporating speculative decoding, that seems like a critical performance optimization,…

> It is also a bit weird that they are not incorporating speculative decoding

Wouldn’t speculative decoding decrease overall throughput, but optimise (perceived) responsiveness?

Re: Surpassing vLLM with a Generated Inference Stack

#16

Post by YetAnotherNick »

jeeeb wrote: post #15
ntonozzi wrote: post #2

Why do they need to run benchmarks to confirm performance? Can't they run an example prompt and verify they get the exact same output token probabilities for all prompts? The fact that they are not doing this makes me suspicious that they are in fact not doing the exact same thing as vLLM. It is also a bit weird that they are not incorporating speculative decoding, that seems like a critical performance optimization,…

> It is also a bit weird that they are not incorporating speculative decoding Wouldn’t speculative decoding decrease overall throughput, but optimise (perceived) responsiveness?

For compute bound region(high batch size) yes, but for low batch size it could improve the throughput.

Re: Surpassing vLLM with a Generated Inference Stack

#19

Post by LuxBennu »

rfw300 wrote: post #11
lukebechtel wrote: post #4

Earlier quoted context omitted.

We validate with MMLU and Hellaswag presently, and are getting this independently verified by a 3rd party. We have considered open-sourcing some of our optimized inference libraries in the future, but have not yet come to a decision on this. Also if you need a rough intuition as to why this is possible: it's because this entire inference stack was built for exactly one model, and thus we can really tune the entire fr…

I've no problem with the intuition. But I would hope for a lot more focus in the marketing materials on proving the (statistical) correctness of the implementation. 15% better inference speed is not worth it to use a completely unknown inference engine not tested across a wide range of generation scenarios.

[flagged]
Post reply on HN