Everyone is building LLM routers, we deprecated ours
manifest.build
Everyone is building LLM routers, we deprecated ours
1–10 of 94 posts
Re: Everyone is building LLM routers, we deprecated ours
#2Ironically, my confidence that a human had at least an active part in writing/editing this article went up because of this train wreck of a sentence:
> "A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it."
Re: Everyone is building LLM routers, we deprecated ours
#3[flagged]
Re: Everyone is building LLM routers, we deprecated ours
#4Ironically, my confidence that a human had at least an active part in writing/editing this article went up because of this train wreck of a sentence: > "A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it."
what’s wrong with the sentence? reads fine to me
Re: Everyone is building LLM routers, we deprecated ours
#5Ironically, my confidence that a human had at least an active part in writing/editing this article went up because of this train wreck of a sentence: > "A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it."
what’s wrong with the sentence? reads fine to me
i think it should be "by adding stickiness to the initially chosen provider of that model"
Re: Everyone is building LLM routers, we deprecated ours
#6Ironically, my confidence that a human had at least an active part in writing/editing this article went up because of this train wreck of a sentence: > "A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it."
what’s wrong with the sentence? reads fine to me
'keeps' is the wrong tense and could be 'will keep'
Re: Everyone is building LLM routers, we deprecated ours
#7Yeah routers suck, been doing this myself for the best part of a year now and it’s really difficult to make it behave. A good model for your task is the best bet.
Re: Everyone is building LLM routers, we deprecated ours
#8Routing belongs on the client side
Re: Everyone is building LLM routers, we deprecated ours
#9I find this routing problem to be opaque and I’m generally skeptical that the label people are trying to predict is meaningful.
If you really need more discrimination of the complexity of an input to get an efficient response, sft or rl tuning something for your harness would be more effective.
Re: Everyone is building LLM routers, we deprecated ours
#10Some routing services not just route to different LLMs, they also handle all the legal issues (GDPR compliance, ISO certification, guaranteed Zero-Data-Retention, domestic data processing/European based clouds, etc.). In regulated industries, these things matter a lot, especially when processing of sensitive data is involved.