41–50 of 58 posts
I think this presentation says more about what knobs you can turn and in what direction the outcome will move (it may be worse than a competitor) than it says about frontiers.
How do you know for sure that you stay on an efficient frontier when you change a parameter? I think this presentation says more about what knobs you can turn and in what direction the outcome will move (it may be worse than a competitor) than it says about frontiers.
this is a nice and concise writeup. what's striking to me is that these techniques really have not changed in /years/. sure, precision has become slightly lower, spec decoding acceptance has gotten slightly better and the complexity of parallelism is trickier with mixture of experts. but no new concepts in a very long time! the absolute most impactful improvements for inference comes at architecture design time. I fi…
"the efficient frontier" is an important landmark of (investment) portfolio theory. It proves/explains/illustrates how you can combine selections from a diffuse cloud of individual investments and still land on a frontier that is better than any of your individual choices. It's the entire basis of "diversify your portfolio". The efficient frontier of LLM inference is a line, not a frontier. this is a frontier: https:…
The efficient frontier of portfolio theory is also a line. Not sure I get your point here.
I'm currently trying to write an inference engine that combines the benefits of llama.cpp (one binary deployment, good support for heterogenous non-datacenter compute, wide quantization support) with the benefits of vLLM/SGlang (things like proper paged attention for better VRAM utilization and high concurrency). Datacenter hardware is expensive and there's shortage of it but llama.cpp is slow/unoptimized for concurr…
Earlier quoted context omitted.
Can we expect similar issues such as spectre and meltdown that intel experienced with speculative execution.. but, in the form of prompt injection/poisoning?
Ok, I'll bite: no, considering these are very different domains and you don't get system access by getting the wrong speculative branch for your next text token, you just get a slightly different (but probably still related enough) text.
Within LLMs, and even inference naturally, there are many other potential parameters that one might optimize: Unsloth typically shows a Pareto frontier for size of a quantized model versus KL divergence. Others trade total concurrent tok/s against single-stream tok/s. KV cache size, context length and context coherency are other trade-offs that are closely related to inference. Total intelligence is usually a defining characteristic of a "frontier model", with cost (per token or task) as a salient trade-off. Cost is one parameter that is implicitly fixed by the "throughput versus latency" analysis: using a GB300 versus Radeon R9700 moves the curve enormously and probably changes the shape of it. Lots of threads here argue over local vs cloud inference regarding cost efficiency, often with privacy and control as competing objectives.
> Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses. As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack. Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design. Then in the mid-2000s the Speculator[0] paper brought that concept to distributed system…
Can we expect similar issues such as spectre and meltdown that intel experienced with speculative execution.. but, in the form of prompt injection/poisoning?
You could possibly in-pronciple detect what a model with a censorship filter was actually saying, but not really. The signal is weak and needs lots of samples to get anything worth having. You just don't have that level of control to set things up.
I'm currently trying to write an inference engine that combines the benefits of llama.cpp (one binary deployment, good support for heterogenous non-datacenter compute, wide quantization support) with the benefits of vLLM/SGlang (things like proper paged attention for better VRAM utilization and high concurrency). Datacenter hardware is expensive and there's shortage of it but llama.cpp is slow/unoptimized for concurr…
I feel like P/D dissaggregation will be the next big one for providers, as prefill tends to be compute bound while decode mem bound which I guess each will have a different type of node