11–20 of 58 posts
this is a nice and concise writeup. what's striking to me is that these techniques really have not changed in /years/. sure, precision has become slightly lower, spec decoding acceptance has gotten slightly better and the complexity of parallelism is trickier with mixture of experts. but no new concepts in a very long time! the absolute most impactful improvements for inference comes at architecture design time. I fi…
this is a nice and concise writeup. what's striking to me is that these techniques really have not changed in /years/. sure, precision has become slightly lower, spec decoding acceptance has gotten slightly better and the complexity of parallelism is trickier with mixture of experts. but no new concepts in a very long time! the absolute most impactful improvements for inference comes at architecture design time. I fi…
I think the biggest net new recent technique is P/D disaggregation. And that spec dec is very different now especially post DSpark/DFlash. But overall yes the fundamentals of LLM performance optimization have been remarkably stable over the last few years.
Inference techniques either move a deployment along the latency–throughput frontier or push the entire frontier out, creating more efficiency to allocate. This is a tautology. You can say that with anything. Gastronomy techniques will make a previous recipe better, or create a new recipe better than others, or a mix of both.
Inference techniques either move a deployment along the latency–throughput frontier or push the entire frontier out, creating more efficiency to allocate. This is a tautology. You can say that with anything. Gastronomy techniques will make a previous recipe better, or create a new recipe better than others, or a mix of both.
The point is to classify them into two kinds. The kind that shifts the frontier is more powerful, since improves capabilities without incurring tradeoffs.
As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack.
Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design.
Then in the mid-2000s the Speculator[0] paper brought that concept to distributed systems, which we’re still seeing work on[1][2].
Everything old is new again (:
[0]https://www.cs.princeton.edu/courses/archive/fall07/cos518/p...
[1]https://www.usenix.org/system/files/osdi25-shen-weihai.pdf
[2] https://www.microsoft.com/en-us/research/publication/distrib...
> Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses. As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack. Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design. Then in the mid-2000s the Speculator[0] paper brought that concept to distributed system…
The efficient frontier of LLM inference is a line, not a frontier.
this is a frontier:
https://upload.wikimedia.org/wikipedia/commons/e/e1/Markowit...
no matter how good is something a smart person writes down, a pleb will come along and try to hang on its coattails.
If you want to steal an idea for this, steal indifference curves, they'd make more sense.
> Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses. As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack. Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design. Then in the mid-2000s the Speculator[0] paper brought that concept to distributed system…
That's because it's just "guess and check" not some deep universal insight.
