Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hypermode Model Router Preview – OpenRouter Alternative

hypermode.com

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Hypermode Model Router Preview – OpenRouter Alternative

#11

Post by iamtherhino »

0xDEAFBEAD wrote: post #5
jbellis wrote: post #2

What I'm seeing with Brokk ( https://brokk.ai ) is that models are not really interchangeable for code authoring. Even with frontier models like GP2.5 and Sonnet 3.7, Sonnet is significantly better about following instructions ("don't add redundant comments") while GP2.5 has more raw intelligence. So we're using litellm to create a unified API to consume but the premise of "route your requests to whatever model is re…

Are there any of these tools which will use your evals to automatically recommend a model to use? Imagine if you didn't need to follow model releases anymore, and you just had a heuristic that would automatically select the right price/performance tradeoff. Maybe there's even a way to route queries differently to more expensive models depending on how tricky they are. (This would be more for using models at scale in…

We've been playing with that in the background. I can try to shoot you a preview in a few weeks. It works pretty well for reasoning tasks/NLP workloads but for workloads that need a "correct" answer, it's really tough to maintain accuracy when swapping models.

What we've seen most successful is making recommendations in the agent creation process for a given tool/workload and then leaving them somewhat static after creation.

Re: Hypermode Model Router Preview – OpenRouter Alternative

#12

Post by 0xDEAFBEAD »

iamtherhino wrote: post #11
0xDEAFBEAD wrote: post #5

Earlier quoted context omitted.

Are there any of these tools which will use your evals to automatically recommend a model to use? Imagine if you didn't need to follow model releases anymore, and you just had a heuristic that would automatically select the right price/performance tradeoff. Maybe there's even a way to route queries differently to more expensive models depending on how tricky they are. (This would be more for using models at scale in…

We've been playing with that in the background. I can try to shoot you a preview in a few weeks. It works pretty well for reasoning tasks/NLP workloads but for workloads that need a "correct" answer, it's really tough to maintain accuracy when swapping models. What we've seen most successful is making recommendations in the agent creation process for a given tool/workload and then leaving them somewhat static after c…

That's fair. Maybe you could even send the user an email if you detect a new model release or pricing change which handles their workload for cheaper at comparable quality, to notify them to investigate.

Re: Hypermode Model Router Preview – OpenRouter Alternative

#13

Post by iamtherhino »

0xDEAFBEAD wrote: post #12
iamtherhino wrote: post #11

Earlier quoted context omitted.

We've been playing with that in the background. I can try to shoot you a preview in a few weeks. It works pretty well for reasoning tasks/NLP workloads but for workloads that need a "correct" answer, it's really tough to maintain accuracy when swapping models. What we've seen most successful is making recommendations in the agent creation process for a given tool/workload and then leaving them somewhat static after c…

That's fair. Maybe you could even send the user an email if you detect a new model release or pricing change which handles their workload for cheaper at comparable quality, to notify them to investigate.

That's a good idea-- then give them a link to "replay last X inferences with model ABC" so they can do a quick eyeball eval.

Re: Hypermode Model Router Preview – OpenRouter Alternative

#14

Post by 0xDEAFBEAD »

jbellis wrote: post #10
0xDEAFBEAD wrote: post #5

Earlier quoted context omitted.

Are there any of these tools which will use your evals to automatically recommend a model to use? Imagine if you didn't need to follow model releases anymore, and you just had a heuristic that would automatically select the right price/performance tradeoff. Maybe there's even a way to route queries differently to more expensive models depending on how tricky they are. (This would be more for using models at scale in…

Yeah, that seems possible, but a dumb preprocessing step won't help and a smart one will add significant latency. Feels a bit halting-problem-ish: can you tell if a problem is too hard for model A without being smarter than model A yourself?

I imagine if your volume is high enough it could be worthwhile to at least check to see if simple preprocessing gets you anywhere.

Basically compare model performance on a bunch of problems, and see if the queries which actually require an expensive model have anything in common (e.g. low Flesch-Kincaid readability, or a bag-of-words approach which tries to detect the frequency of subordinate clauses/potentially ambiguous pronouns, or word rarity, or whatever).

Maybe my knowledge of old-school NLP methods is useful after all :-) Generally those methods tend to be far less compute-intensive. If you wanted to go really crazy on performance, you might even use a Bloom filter to do fast, imprecise counting of words of various types.

Then you could add some old-school, compute-lite ML, like an ordinary linear regression on the old-school-NLP-derived features.

Really the win would be for a company like Hypermode to implement this automatically for customers who want it (high volume customers who don't mind saving money).

Actually, a company like Hypermode might be uniquely well-positioned to offer this service to smaller customers as well, if query difficulty heuristics generalize well across different workloads. Assuming they have access to data for a large variety of customers, they could look for heuristics that generalize well.

Re: Hypermode Model Router Preview – OpenRouter Alternative

#15

Post by 0xDEAFBEAD »

iamtherhino wrote: post #13
0xDEAFBEAD wrote: post #12

Earlier quoted context omitted.

That's fair. Maybe you could even send the user an email if you detect a new model release or pricing change which handles their workload for cheaper at comparable quality, to notify them to investigate.

That's a good idea-- then give them a link to "replay last X inferences with model ABC" so they can do a quick eyeball eval.

Sweet, maybe you'll like my other idea in this thread too: https://news.ycombinator.com/item?id=43929194

Re: Hypermode Model Router Preview – OpenRouter Alternative

#16

Post by iamtherhino »

0xDEAFBEAD wrote: post #14
jbellis wrote: post #10

Earlier quoted context omitted.

Yeah, that seems possible, but a dumb preprocessing step won't help and a smart one will add significant latency. Feels a bit halting-problem-ish: can you tell if a problem is too hard for model A without being smarter than model A yourself?

I imagine if your volume is high enough it could be worthwhile to at least check to see if simple preprocessing gets you anywhere. Basically compare model performance on a bunch of problems, and see if the queries which actually require an expensive model have anything in common (e.g. low Flesch-Kincaid readability, or a bag-of-words approach which tries to detect the frequency of subordinate clauses/potentially ambi…

I really like this approach.

I think there's a big advantage to be had for folks brining "old school" ML approaches to LLMs. We've been spending a lot of time looking at the expert systems from the 90s.

Another one we've been looking at is applying some query planning approaches to these systems to see if we can pull responses from cache instead of invoking the model again.

Obviously there's a lot of complexity to identifying where we could apply some smaller ML models or cache-- but it's been a really fun exploration.

Re: Hypermode Model Router Preview – OpenRouter Alternative

#19

Post by 0xDEAFBEAD »

iamtherhino wrote: post #16
0xDEAFBEAD wrote: post #14

Earlier quoted context omitted.

I imagine if your volume is high enough it could be worthwhile to at least check to see if simple preprocessing gets you anywhere. Basically compare model performance on a bunch of problems, and see if the queries which actually require an expensive model have anything in common (e.g. low Flesch-Kincaid readability, or a bag-of-words approach which tries to detect the frequency of subordinate clauses/potentially ambi…

I really like this approach. I think there's a big advantage to be had for folks brining "old school" ML approaches to LLMs. We've been spending a lot of time looking at the expert systems from the 90s. Another one we've been looking at is applying some query planning approaches to these systems to see if we can pull responses from cache instead of invoking the model again. Obviously there's a lot of complexity to id…

>We've been spending a lot of time looking at the expert systems from the 90s.

No way. I would definitely be curious to hear more if you want to share.

Re: Hypermode Model Router Preview – OpenRouter Alternative

#20

Post by iamtherhino »

threeducks wrote: post #6

The Python API example looks like it has been written by an LLM. You don't need to import json, you don't need to set the content type and it is good practice to use context managers ("with" statement) to release the connection in case of exceptions. Also, you don't gain anything by commenting variables with the name of the variable. The following sample (probably) does the same thing and is almost half as short. I h…

updated our python example too!
Post reply on HN