Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everyone is building LLM routers, we deprecated ours

manifest.build

Post reply on HN
Print view View original

61–70 of 94 posts

Re: Everyone is building LLM routers, we deprecated ours

#61

Post by overgard »

danielmarkbruce wrote: post #44
overgard wrote: post #12

> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…

You don't just try every model that comes along... and for the ones you do, you have eval pipelines set up. It's engineering, you have to test stuff works. Now, there are thousands of use cases for lots of the fontier models. I don't use them for everything, so I don't test them except on the stuff I'm using them for.

Well, first off, I don't need to test that stuff works, the labs need to do that! I can afford to wait a couple days to get an impression on how people like a model before I switch. I don't want to be a beta tester.

But honestly, even if I went with your idea of picking a small number of models to eval, I wouldn't even know how to choose that subset other than.. what's hottest in the hype cycle? The benchmarks are useless and gamed. There's not a lot of transparency in how they're trained, so there's no way for me to go like "Well, I care about how much graphics programming it's been trained on, and I can see X ingested Y amount of training data on that.." It's basically a stab in the dark of very frothy people saying "X CHANGES EVERYTHING!" with no particular details other than a bunch of guys on youtube being like "I gave it a really lame prompt and it made me this game that absolutely nobody would ever want to play!"

Re: Everyone is building LLM routers, we deprecated ours

#63

Post by kaycey2022 »

It seems apparent to me that task complexity can’t be determined by prompt alone. How? A prompt is just a simple rambling. An agent will go through many many tool calls and steering just to arrive at the right approach.

A serious router therefore needs to build up a dataset of how different models responded, end to end, to different tasks on different contexts. I won’t comment on whether the current frontier models can reliably judge these outputs, but I am sceptical about that. And moreover look at the state of evils! They are gamed to hell and keep losing credibility.

A more difficult problem for router builders is that they are working on an opaque system behind an external API. How can you reliably guarantee model behaviour when model behaviour has been shown to deteriorate under arbitrary conditions that have nothing to do with the task given? So much investment only to be an AI company that can get rug pulled by the real AI companies at any given time.

I would think the only people who can come up with good routers for a collection of models are the inference providers themselves. Because theoretically they have full control of how their models are served. And inference is not zero cost or cheap for them either. And going by OpenAI’s experience routing is not an easy problem for them to solve either. And they don’t have the incentive to route you to cheaper models and reduce costs for customers at the same time. Routing objective for them is to increase their own profits.

Re: Everyone is building LLM routers, we deprecated ours

#64

Post by rrvsh »

Didn't this slightly contradict itself? The conclusion I drew based on the problem statement was that the best way was to have a model router handle the root prompt and stick to a model for the rest of the session - engineers still have the discretion to handoff to a smaller or larger model when needs arise.

Re: Everyone is building LLM routers, we deprecated ours

#65

Post by drusepth »

overgard wrote: post #12

> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…

One of the best features I built into our game studio's AI router (IMO) was to take a random sampling of generations and have them generate across ALL models (that we support, at least) so we, the developers, can browse outputs and get a sense each model's output. Seeing them all side by side for generations you're already familiar with makes it feel like significantly less cognitive load.

The better I get at recognizing what each model is good/bad at, the more I'm glad we're taking the time to choose specific models for specific prompts -- and the more I wouldn't trust a generic router to efficiently route for me.

Re: Everyone is building LLM routers, we deprecated ours

#66

Post by brunaxLorax »

maherbeg wrote: post #41

Now that even the smaller models from labs (Luna, deepseek v4 flash) are getting powerful, I think the orchestrator pattern of a smart model coordinating smaller models for work will end up being the way to go.

Agree. The orchestrator pattern (big model managing smaller ones) works already well

Re: Everyone is building LLM routers, we deprecated ours

#67

Post by brunaxLorax »

coffinbirth wrote: post #10

Some routing services not just route to different LLMs, they also handle all the legal issues (GDPR compliance, ISO certification, guaranteed Zero-Data-Retention, domestic data processing/European based clouds, etc.). In regulated industries, these things matter a lot, especially when processing of sensitive data is involved.

If your router has GDPR/ZDR/EU compliant, it doesn't make your provider compliant. You're just adding a middleman.

Re: Everyone is building LLM routers, we deprecated ours

#68

Post by brunaxLorax »

seizethecheese wrote: post #19

Their router classified prompts into difficulty buckets. This obviously won't work. Consider a senior developer routing work based only on the task description. Clearly you need to dig a bit deeper into the task. Saying routers don't work is sort of like saying serverless doesn't work. It depends on when and how! One routing implementation that recently launched here is interesting ( https://news.ycombinator.com/item…

I didn't know about Tokenless, the approach seems really innovative, if it works it fixes the "Complexity cannot be deduced from the prompt alone" problem. However you still have the other hidden costs: cache, breaking behavior consistency, and unpredictability.

Your point is interesting, you say that task specificity is easier to classify than task complexity, which I agree - I didn't mention it but we had task specificity routing too :). My opinion is that in many cases task specificity calls are easy to distinguish at build time, and therefore you can isolate them and attach the right model/settings beforehand, so you have less need for a smart routing on the fly.

Re: Everyone is building LLM routers, we deprecated ours

#69

Post by brunaxLorax »

ramraj07 wrote: post #15
robertclaus wrote: post #13

I think fail-over for a production critical service is an equally important responsibility of the router.

You cant fail over to a different model though. Even failing over to a different provider isn't always as simple as set it and forget it. For example bedrock doesn't support the tool search tool feature from anthropic.

I didn't know that! I thought that all models had the same features no matter the provider
Post reply on HN