My daily struggle is trying to make a 7B model respond in under 500ms without breaking the bank. This hits home.
The efficient frontier of LLM inference
31–40 of 57 posts
Re: The efficient frontier of LLM inference
#32> Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses. As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack. Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design. Then in the mid-2000s the Speculator[0] paper brought that concept to distributed system…
> optimizations applied at different levels of the stack That's because it's just "guess and check" not some deep universal insight.
It's closer to "we can make this highly parallel for not that much cost, but we struggle to use that concurrency. So what if we just guess what the next step is going to be? If we are right we get a big speedup, if we are wrong we just throw that work away". Which I would classify as a notable insight. Doing work that you are 50% certain is useless is not the most obvious thing
Re: The efficient frontier of LLM inference
#33My RTX 3090 is still laughing at my attempts to run 70B models efficiently.
Re: The efficient frontier of LLM inference
#34[dead]
Re: The efficient frontier of LLM inference
#35[flagged]
Re: The efficient frontier of LLM inference
#36I'm currently trying to write an inference engine that combines the benefits of llama.cpp (one binary deployment, good support for heterogenous non-datacenter compute, wide quantization support) with the benefits of vLLM/SGlang (things like proper paged attention for better VRAM utilization and high concurrency).
Datacenter hardware is expensive and there's shortage of it but llama.cpp is slow/unoptimized for concurrent use, while vLLM/SGLang easily crash on non-common setups (things like, if you do pipeline parallelism for RTX5090+RTX4090, they will randomly crash with RAM caching enabled or select wrong kernels because they usually assume that every rank is the same device type; they also don't support Q5-Q6).
For me what's most interesting is to optimize inference for lack of good datacenter hardware and how to optimize for it best. I've been running an AI server in the office, and so far I've find these techniques most important for concurrent use on cheap hardware: pipeline parallelism (to accomodate for PCie), RAM caching (to quickly restore contexts into VRAM), speculative decoding (including domain-specific ngrams, they already can speed up code generation considerably without the overhead of a draft model), good kernels highly optimized for a specific device, support for Q5-Q6 (almost as good as Q8), FP8 contexts (more context to fit), paged attention (for better VRAM utilization), prefix caching, continuous batching (this is the default everywhere).
So far the main bottlenecks have been llama.cpp's poor VRAM utilization for contexts (you either have fixed-size slots, or use unified KV cache where each request attends to attention from all other requests and then unnecessary portions of attention are masked out), and lack of decode/prefill segregation: when a request starts prefilling a long context, all decoding threads slow down to like 5 tok/sec. On the other hand, vLLM/SGLang feel superbuggy if you don't run them on some officially approved node like 8xH200
Re: The efficient frontier of LLM inference
#37I think we can soon include "recursive depth" strategy that Astra is employing, which (I suspect) is using recursive internal state changes in the transformer as opposed to full forward-pass + sampling which has traditionally been the case with thinking/CoT. Similar method was used here (but different context - encoding tools inside the transformer weights for fast execution): https://www.percepta.ai/blog/can-llms-be-computers
Re: The efficient frontier of LLM inference
#38"the efficient frontier" is an important landmark of (investment) portfolio theory. It proves/explains/illustrates how you can combine selections from a diffuse cloud of individual investments and still land on a frontier that is better than any of your individual choices. It's the entire basis of "diversify your portfolio". The efficient frontier of LLM inference is a line, not a frontier. this is a frontier: https:…
The authors are referring to a Pareto Frontier https://en.wikipedia.org/wiki/Pareto_front
Re: The efficient frontier of LLM inference
#39Quantization and speculative decoding unlocked major savings for our smaller models. Still chasing that ideal cost/performance ratio.
Re: The efficient frontier of LLM inference
#40[flagged]