Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everyone is building LLM routers, we deprecated ours

manifest.build

Post reply on HN
Print view View original

81–90 of 94 posts

Re: Everyone is building LLM routers, we deprecated ours

#81

Post by cyanregiment »

overgard wrote: post #60
cyanregiment wrote: post #53

Earlier quoted context omitted.

There are "Hi" and "Lo" variants of the different models. Maybe Gemini or ChatGPT is a "Hi" while running a local Mistral 7b is a "Lo" - sometimes you want lo-res technologies. Faster, lighter, does the job. Find the same with image, video, vision, and so-on. Probably get away with pretty weak vision models for some basic edge detection, but need "a good one" for things like OCR. I don't find a ton of competitive dif…

I concur that there isn't that much difference between the good ones. I'm not really trying to push the limits of what they can do or anything, but I'd say on average I give it tasks about the size of something that would take me a half day to do by hand, and honestly, other than minor quirks in coding style and word choices, they've all basically been "good enough" for what I want. When I tried fable it just wasn't…

Yeah definitely good enough for work. I think people only get into trouble going “agentic” and trying to automate je ne sais quoi

Re: Everyone is building LLM routers, we deprecated ours

#82

Post by mrkeen »

overgard wrote: post #12

> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…

A programmer is someone who declares "I understand the problem so well that I can teach a computer to solve the problem, even for examples I have never seen before".

If it's too hard for a programmer to reason about, I wouldn't trust the programmer sitting next to him to automate the problem away.

Re: Everyone is building LLM routers, we deprecated ours

#83

Post by fennecbutt »

drusepth wrote: post #65
overgard wrote: post #12

> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…

One of the best features I built into our game studio's AI router (IMO) was to take a random sampling of generations and have them generate across ALL models (that we support, at least) so we, the developers, can browse outputs and get a sense each model's output. Seeing them all side by side for generations you're already familiar with makes it feel like significantly less cognitive load. The better I get at recogni…

This is a problem we're looking at atm. We want to start running evaluations as we're having problems with consistency across releases even just for individual agents.

Re: Everyone is building LLM routers, we deprecated ours

#84

Post by iLoveOncall »

overgard wrote: post #12

> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…

Yeah I'm growing increasingly tired of those stupid metaphors about AI.

LLMs are not like screwdrivers, IDEs, or any other tool. They're indeterministic.

I keep seeing everyone make wilder and wilder metaphors which just don't work at all, as a sort of "argument" supporting whatever they say, with 0 actual proof.

Re: Everyone is building LLM routers, we deprecated ours

#85

Post by paradoxyl »

iLoveOncall wrote: post #84
overgard wrote: post #12

> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…

Yeah I'm growing increasingly tired of those stupid metaphors about AI. LLMs are not like screwdrivers, IDEs, or any other tool. They're indeterministic. I keep seeing everyone make wilder and wilder metaphors which just don't work at all, as a sort of "argument" supporting whatever they say, with 0 actual proof.

AIs are like (insert clickbait keyword). You just have to (insert clickbait call to action).

Re: Everyone is building LLM routers, we deprecated ours

#86

Post by TZubiri »

overgard wrote: post #12

> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…

I'm all for dropping priority on vendor neutrality and tightly coupling with vendors.

But model aggregators have a layman smell to me, some laymen think model A talking to model B is materially different than model A talking to model A with another context, also laymen tend to see 'learning AI' as trying out new products and keeping up with product relases.

I think it's less like having a toolbox of screwdrivers, hammers and a wrench, and more like having 30 brands of shovels because there's been a wave of shovel commercials.

Re: Everyone is building LLM routers, we deprecated ours

#87

Post by TZubiri »

dweez wrote: post #18

I spent a lot of time researching LLM routing last year and also came to the conclusion that it's generally not worth the effort. It's too hard to understand the difficulty of a query a priori. One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" ( https://en.wikipedia.org/wiki/Busy_beaver…

How does one spend a lot of time researching model routing? Did you evaluate different routing providers? Maybe you could build a rorouterter that lets you easily switch to any router, ideally if there's a vuln in any of them you can get infected even if you don't use it.

Re: Everyone is building LLM routers, we deprecated ours

#88

Post by TZubiri »

zbentley wrote: post #25
dofm wrote: post #23

Earlier quoted context omitted.

You also have problems with short prompts whose results depend highly on the understanding of nuance. The router is going to have to mostly solve the prompt to decide which model to send it to. What you actually want is a model that can conclude either "I know the answer to this with confidence" and answer, or "I think I don't know the answer to this, I should ask another model and I know which one". But I don't thin…

> Their internal confidence can be measured and returned It can? I was under the impression that confidence was either self-reported by the LLM or assessed by having another model interpret the output response. If there's a confidence score at the level of the actual model math, that's news to me.

Logprobs

Re: Everyone is building LLM routers, we deprecated ours

#89

Post by jurgenburgen »

overgard wrote: post #12

> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…

There’s a heuristic I’ve been using to pick which model to use. The provider is already chosen by my org so what I do is activate the model picker in my harness, select the most expensive model’s latest version and set it on the highest effort setting.

This has worked quite well for me and saves a lot of time.

Re: Everyone is building LLM routers, we deprecated ours

#90

Post by danielmarkbruce »

overgard wrote: post #61
danielmarkbruce wrote: post #44

Earlier quoted context omitted.

You don't just try every model that comes along... and for the ones you do, you have eval pipelines set up. It's engineering, you have to test stuff works. Now, there are thousands of use cases for lots of the fontier models. I don't use them for everything, so I don't test them except on the stuff I'm using them for.

Well, first off, I don't need to test that stuff works, the labs need to do that! I can afford to wait a couple days to get an impression on how people like a model before I switch. I don't want to be a beta tester. But honestly, even if I went with your idea of picking a small number of models to eval, I wouldn't even know how to choose that subset other than.. what's hottest in the hype cycle? The benchmarks are us…

You are completely wrong and will never make a good product/service with that attitude. You need to test the models work for your use case.

Knowing how much of type X data a model was trained on isn't going to help you one bit. Various phases of training can effectively wipe out the training from earlier steps if not done well. All that matters is does the model do what you want it to do. In practice almost everything in AI/ML is empirical.

Post reply on HN