Classifying complexity from the prompt alone was always the weak point. Most of what makes a task hard shows up after the first tool call.
Everyone is building LLM routers, we deprecated ours
51–60 of 94 posts
Re: Everyone is building LLM routers, we deprecated ours
#52[flagged]
Re: Everyone is building LLM routers, we deprecated ours
#53> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…
There are "Hi" and "Lo" variants of the different models.
Maybe Gemini or ChatGPT is a "Hi" while running a local Mistral 7b is a "Lo" - sometimes you want lo-res technologies. Faster, lighter, does the job.
Find the same with image, video, vision, and so-on.
Probably get away with pretty weak vision models for some basic edge detection, but need "a good one" for things like OCR.
I don't find a ton of competitive difference amongst the "good ones" especially not reflected in the pricing and marketing. IMO there are pretty much just "non-AI" algorithms, "basic AI" that are getting more interesting on-device for some use cases, and "good AI" or what we'd call "frontier models" - which again, not seeing a ton of differentiation on that layer.
Re: Everyone is building LLM routers, we deprecated ours
#54Earlier quoted context omitted.
I just use whatever's cheapest for personal things. Dsv4 flash and got 5.6 Luna are great for the price and I'll use the openrouter benchmark view to compare the overall score.
Found the drywaller
My landlord just muds over cob webs and open space. They're over 100 years old so they legally count as part of the building's foundation.
So is he, and so does he. He looks like the Tales From The Crypt guy I have no idea how he's still doing manual labor.
Re: Everyone is building LLM routers, we deprecated ours
#55[flagged]
Re: Everyone is building LLM routers, we deprecated ours
#56> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…
There are "Hi" and "Lo" variants of the different models. Maybe Gemini or ChatGPT is a "Hi" while running a local Mistral 7b is a "Lo" - sometimes you want lo-res technologies. Faster, lighter, does the job. Find the same with image, video, vision, and so-on. Probably get away with pretty weak vision models for some basic edge detection, but need "a good one" for things like OCR. I don't find a ton of competitive dif…
I feel like nearly anything done by a "Lo" is better done myself. I use "Hi" for pushing the limits of my expertise.
Re: Everyone is building LLM routers, we deprecated ours
#57> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…
I just ask our fleet maintenance agent to configure the variants of the new model and then run the bakeoff. In a few hours, I get a doc containing exactly how it performed on my real tasks, against the other models we've benchmarked.
We don't do any "prompt engineering" to make specific models work better or worse, so this gives us a direct comparison that lets us objectively compare how they'll work in practice for different types of tasks.
Good enough to have guided multiple model swaps and measured increases in productivity.
Re: Everyone is building LLM routers, we deprecated ours
#58every LLM router eventually reduces to a config file that says "use the good one" and nobody can define what good means
Re: Everyone is building LLM routers, we deprecated ours
#59Earlier quoted context omitted.
There are "Hi" and "Lo" variants of the different models. Maybe Gemini or ChatGPT is a "Hi" while running a local Mistral 7b is a "Lo" - sometimes you want lo-res technologies. Faster, lighter, does the job. Find the same with image, video, vision, and so-on. Probably get away with pretty weak vision models for some basic edge detection, but need "a good one" for things like OCR. I don't find a ton of competitive dif…
I feel like nearly anything done by a "Lo" is better done myself. I use "Hi" for pushing the limits of my expertise.
A good use case for me is automating invoice notes that are all very similar, but change based on details of client and work. Mistral 7b does it fine, instantly.
UX-wise, I don't predict out "ghost text" like Google has been doing, I just preload the input value with the prediction. You can edit as you want. Saves a ton of time.
Re: Everyone is building LLM routers, we deprecated ours
#60> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…
There are "Hi" and "Lo" variants of the different models. Maybe Gemini or ChatGPT is a "Hi" while running a local Mistral 7b is a "Lo" - sometimes you want lo-res technologies. Faster, lighter, does the job. Find the same with image, video, vision, and so-on. Probably get away with pretty weak vision models for some basic edge detection, but need "a good one" for things like OCR. I don't find a ton of competitive dif…
I concur that there isn't that much difference between the good ones. I'm not really trying to push the limits of what they can do or anything, but I'd say on average I give it tasks about the size of something that would take me a half day to do by hand, and honestly, other than minor quirks in coding style and word choices, they've all basically been "good enough" for what I want. When I tried fable it just wasn't for me because it was basically very expensive overkill. I usually stay on the older models as long as I can just because they're predictable and I care a lot more about predictability and cost than being on the super cutting edge or whatever. (Honestly I'd kind of rather the enthusiasts test it first for the various "new model deleted my database" bugs that seem to pop up.