Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing
1–10 of 29 posts
Re: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing
#2Wow, that was fast.
I've thought for a while that ensembling approaches would become the next stage of LLM development after CoT, since it provides yet another effective, independent axis for scaling laws. Great to see that perspective is taking off. The open weight community has an opportunity to take these ideas and run with them better than OpenAI has.
Re: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing
#3I’m fascinated by this new paradigm. We’ve more or less perfected Mixture-of-Experts inside a single model, where routing happens between subnetworks. What GPT-5 auto (and this paper) are doing is a step further: “LLM routing” across multiple distinct models. It’s still rough right now, but it feels inevitable that this will get much better over time.
Re: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing
#4I’m fascinated by this new paradigm. We’ve more or less perfected Mixture-of-Experts inside a single model, where routing happens between subnetworks. What GPT-5 auto (and this paper) are doing is a step further: “LLM routing” across multiple distinct models. It’s still rough right now, but it feels inevitable that this will get much better over time.
> It’s still rough right now, but it feels inevitable that this will get much better over time.
Yeah, the signals they get will improve things over time. You can do a lot of heavy lifting with embedding models nowadays, get "satisfaction" signals from chats, and adjust your router based on those. It will be weird at first, some people will complain, but at the end of the day, you don't need imo-gold levels of thinking to write a fitness plan that most likely the user won't even follow :)
Signal gathering is likely the driver of most of the subsidised model offerings we see today.
Re: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing
#5Paper and repo do not mention routing latency, which I think is a concern.
Also the paper has some pie chart crimes on page 6.
Re: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing
#6Paper and repo do not mention routing latency, which I think is a concern. Also the paper has some pie chart crimes on page 6.
Just from a brief look at the repo they seem to be doing semantic embeddings w/ Qwen3-Embedding-8B, which should be in the high thousands pp t/s on recent hardware. With a sufficiently large dataset after using it for a while you could probably fine-tune a smaller model as well (4B and 0.6B available from the same family)
Re: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing
#7Between these kinds of optimizations, improved data center efficiency, and smaller models being more capable, I wonder how long it will be before someone manages to make a profitable AI business. Maybe when they race to train better models slows down and they don't need to constantly upgrade capacity.
Re: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing
#8Between these kinds of optimizations, improved data center efficiency, and smaller models being more capable, I wonder how long it will be before someone manages to make a profitable AI business. Maybe when they race to train better models slows down and they don't need to constantly upgrade capacity.
Reminds me of the early days of cloud computing. It was very pricey, but once the tools caught up in 5 or so years, it went from "omg cloud is so expensive" to "omg cloud is only expensive when its worth building your own data center"
Re: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing
#9Link to repo for those interested: https://github.com/ZhangYiqun018/AvengersPro
Re: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing
#10Between these kinds of optimizations, improved data center efficiency, and smaller models being more capable, I wonder how long it will be before someone manages to make a profitable AI business. Maybe when they race to train better models slows down and they don't need to constantly upgrade capacity.
AGI will not be a single model. It will be an ensemble of models that interact with each other. Just like different parts of your brain.