[flagged]
The efficient frontier of LLM inference
21–30 of 58 posts
Re: The efficient frontier of LLM inference
#22> Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses. As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack. Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design. Then in the mid-2000s the Speculator[0] paper brought that concept to distributed system…
Can we expect similar issues such as spectre and meltdown that intel experienced with speculative execution.. but, in the form of prompt injection/poisoning?
Ok, I'll bite: no, considering these are very different domains and you don't get system access by getting the wrong speculative branch for your next text token, you just get a slightly different (but probably still related enough) text.
Re: The efficient frontier of LLM inference
#23[dead]
Re: The efficient frontier of LLM inference
#24You know what I'm curious about? Whether you have brand guidelines inside the company, a Claude skillset, or the blog post author makes the charts in line with the brand colours and so on.
I draw my diagrams on notecards and send them to our designer who brings them to life.
The images start out looking like this: https://philipkiely.com/images/blogs/how-to-write-a-book/des...
Re: The efficient frontier of LLM inference
#25Nice read. I was wondering what can one do to get into inference engineering as simple theoretical knowledge is not sufficient and switching profiles is tough for someone with years of experience.
Re: The efficient frontier of LLM inference
#26> A model is a “frontier model” if it offers the highest degree of intelligence at a given cost or size.
I would define a "frontier model" as offering the highest degree of intelligence at any cost, or without regard to cost. The frontier today is clearly Fable/Mythos, with the "efficient frontier" at Opus/Sol.
Re: The efficient frontier of LLM inference
#27You know what I'm curious about? Whether you have brand guidelines inside the company, a Claude skillset, or the blog post author makes the charts in line with the brand colours and so on.
I draw my diagrams on notecards and send them to our designer who brings them to life. The images start out looking like this: https://philipkiely.com/images/blogs/how-to-write-a-book/des...
Makes perfect sense. The classic way!
Re: The efficient frontier of LLM inference
#28> A model is a “frontier model” if it offers the highest degree of intelligence at a given cost or size. I would define a "frontier model" as offering the highest degree of intelligence at any cost, or without regard to cost. The frontier today is clearly Fable/Mythos, with the "efficient frontier" at Opus/Sol.
Your partial quote is quite misleading. The article obviously talks about "efficient frontier", not "intelligent frontier".
>In the AI industry, we borrowed the term “efficient frontier” from economists. We use it to talk about managing tradeoffs, most often the tradeoff between cost and capabilities for models. A model is a “frontier model” if it offers the highest degree of intelligence at a given cost or size.
Re: The efficient frontier of LLM inference
#29> Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses. As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack. Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design. Then in the mid-2000s the Speculator[0] paper brought that concept to distributed system…
> optimizations applied at different levels of the stack That's because it's just "guess and check" not some deep universal insight.
Simple optimizations are often the best :)
Re: The efficient frontier of LLM inference
#30> Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses. As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack. Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design. Then in the mid-2000s the Speculator[0] paper brought that concept to distributed system…
> guessing which tokens a model might generate
A transformer that generates tokens based on pre-training? We could call it a GPT for short.