What's the jitter what's the std? What about 1:1 output equality? What's the post request latency of this part? What the ftt?
Good questions! It's clear I need to gather more metrics from our next generated inference library.
Surpassing vLLM with a Generated Inference Stack
21–26 of 26 posts
Re: Surpassing vLLM with a Generated Inference Stack
#22Every example like this makes it obvious that you can now use ML-like optimization approaches on well-specified, very-well-tested software problems with a clear optimization goal. Keep if it improves the objective while maintaining correctness, discard if it doesn't. AI-descent strikes again. Maybe I should learn more about ML to have a better instinct on optimization methods in general, so I can actually build AI op…
The bitter lesson strikes again, I suppose!
Re: Surpassing vLLM with a Generated Inference Stack
#23Any place we can find the code?
Unfortunately it hasn't been open sourced. We're debating how / when to do this right now.
Confusing, since this is specific to an architecture that no one making money will use (8B is consumer space, not enterprise).
The produced code shouldn't hold much interesting IP?
Re: Surpassing vLLM with a Generated Inference Stack
#24Why do they need to run benchmarks to confirm performance? Can't they run an example prompt and verify they get the exact same output token probabilities for all prompts? The fact that they are not doing this makes me suspicious that they are in fact not doing the exact same thing as vLLM. It is also a bit weird that they are not incorporating speculative decoding, that seems like a critical performance optimization,…
> Can't they run an example prompt and verify they get the exact same output token probabilities for all prompts?
You don’t even get that with GPUs in general, or really floating point in general.
The Art of Computer Programming. Volume 2: Seminumerical Algorithms section 4.2.2 with explain where it loses floating addition associativity property.
Apartness relations are another possible lens.
Re: Surpassing vLLM with a Generated Inference Stack
#25Why do they need to run benchmarks to confirm performance? Can't they run an example prompt and verify they get the exact same output token probabilities for all prompts? The fact that they are not doing this makes me suspicious that they are in fact not doing the exact same thing as vLLM. It is also a bit weird that they are not incorporating speculative decoding, that seems like a critical performance optimization,…
> Can't they run an example prompt and verify they get the exact same output token probabilities for all prompts? You don’t even get that with GPUs in general, or really floating point in general. The Art of Computer Programming. Volume 2: Seminumerical Algorithms section 4.2.2 with explain where it loses floating addition associativity property. Apartness relations are another possible lens.
Re: Surpassing vLLM with a Generated Inference Stack
#26Earlier quoted context omitted.
> Can't they run an example prompt and verify they get the exact same output token probabilities for all prompts? You don’t even get that with GPUs in general, or really floating point in general. The Art of Computer Programming. Volume 2: Seminumerical Algorithms section 4.2.2 with explain where it loses floating addition associativity property. Apartness relations are another possible lens.
Yeah you can: https://thinkingmachines.ai/blog/defeating-nondeterminism-in... .
> However, as the name “batch-invariant” suggests, the technique is currently limited to handling variations related only to the batch dimension, making it robust to continuous batching and other batch-size–related changes, but not to other forms of nondeterminism like changing the TP sizes or GPU types.