Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everyone is building LLM routers, we deprecated ours

manifest.build

Post reply on HN
Print view View original

21–30 of 94 posts

Re: Everyone is building LLM routers, we deprecated ours

#21

Post by zackify »

overgard wrote: post #12

> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…

I just use whatever's cheapest for personal things. Dsv4 flash and got 5.6 Luna are great for the price and I'll use the openrouter benchmark view to compare the overall score.

Re: Everyone is building LLM routers, we deprecated ours

#22

Post by bluejay2387 »

I think this should probably be scoped to 'generic router systems that don't understand query context' are not useful. We have had lots of good results with routers that understand the context of the types of workloads they process and can route requests to the most efficient models.

Re: Everyone is building LLM routers, we deprecated ours

#23

Post by dofm »

dweez wrote: post #18

I spent a lot of time researching LLM routing last year and also came to the conclusion that it's generally not worth the effort. It's too hard to understand the difficulty of a query a priori. One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" ( https://en.wikipedia.org/wiki/Busy_beaver…

You also have problems with short prompts whose results depend highly on the understanding of nuance. The router is going to have to mostly solve the prompt to decide which model to send it to.

What you actually want is a model that can conclude either "I know the answer to this with confidence" and answer, or "I think I don't know the answer to this, I should ask another model and I know which one". But I don't think LLMs can really bring their uncertainty to the surface in that way yet? Their internal confidence can be measured and returned, so you could probably front a more powerful model with a knowledgeable assistant, but they can't consciously mark their own homework?

Re: Everyone is building LLM routers, we deprecated ours

#24

Post by try-working »

I recently wrote about first principles of model routing that I've learnt building a model router.

The model pool should be kept small, and models in the pool should be clearly differentiated. For example, one large frontier model for quality, one small, fast and cheap model like DeepSeek V4 Flash for routing work.

These two principles by themselves solve the issues with caching, with routing decision making. I routinely hit >99% cache while routing between GPT 5.4 and DeepSeek.

https://try.works/first-principles-of-model-routing

Re: Everyone is building LLM routers, we deprecated ours

#25

Post by zbentley »

dofm wrote: post #23
dweez wrote: post #18

I spent a lot of time researching LLM routing last year and also came to the conclusion that it's generally not worth the effort. It's too hard to understand the difficulty of a query a priori. One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" ( https://en.wikipedia.org/wiki/Busy_beaver…

You also have problems with short prompts whose results depend highly on the understanding of nuance. The router is going to have to mostly solve the prompt to decide which model to send it to. What you actually want is a model that can conclude either "I know the answer to this with confidence" and answer, or "I think I don't know the answer to this, I should ask another model and I know which one". But I don't thin…

> Their internal confidence can be measured and returned

It can? I was under the impression that confidence was either self-reported by the LLM or assessed by having another model interpret the output response. If there's a confidence score at the level of the actual model math, that's news to me.

Re: Everyone is building LLM routers, we deprecated ours

#26

Post by mountainriver »

overgard wrote: post #12

> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…

You do still need to know though, different prompts have different affects on them.

Re: Everyone is building LLM routers, we deprecated ours

#27

Post by bobmcnamara »

zackify wrote: post #21
overgard wrote: post #12

> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…

I just use whatever's cheapest for personal things. Dsv4 flash and got 5.6 Luna are great for the price and I'll use the openrouter benchmark view to compare the overall score.

Found the drywaller

Re: Everyone is building LLM routers, we deprecated ours

#29

Post by jdiff »

zbentley wrote: post #25
dofm wrote: post #23

Earlier quoted context omitted.

You also have problems with short prompts whose results depend highly on the understanding of nuance. The router is going to have to mostly solve the prompt to decide which model to send it to. What you actually want is a model that can conclude either "I know the answer to this with confidence" and answer, or "I think I don't know the answer to this, I should ask another model and I know which one". But I don't thin…

> Their internal confidence can be measured and returned It can? I was under the impression that confidence was either self-reported by the LLM or assessed by having another model interpret the output response. If there's a confidence score at the level of the actual model math, that's news to me.

At least one model can today. https://github.com/cactus-compute/cactus-hybrid

Re: Everyone is building LLM routers, we deprecated ours

#30

Post by 0xDEAFBEAD »

From a big-O notation perspective, if you're serving N queries, working the kinks out of your routing system costs perhaps O(logN) in developer time, whereas LLM provider savings grow as O(N).

In other words, the more queries you're serving, the more worthwhile it looks to figure out a viable method of model routing.

Post reply on HN