Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Tokenless (YC S26) – Automatic model switching to save money

usetokenless.com

Post reply on HN
Print view View original

31–40 of 69 posts

Re: Launch HN: Tokenless (YC S26) – Automatic model switching to save money

#31

Post by brandall10 »

verdverm wrote: post #29
brandall10 wrote: post #27

Earlier quoted context omitted.

Seems like it might be more advantageous to just adjust reasoning effort to retain cache. Maybe in some exceptional cases where there will be a ton more inference to solve the problem, but going significantly dumber in that case seems counterintuitive. I can really only see the utility of things like spawning subagents to a lower tier model from another provider, and that's something harnesses can already handle (ie.…

caching is per model, it does not transfer between them

Updated to clarify reasoning effort.

Re: Launch HN: Tokenless (YC S26) – Automatic model switching to save money

#33

Post by rohaga »

brandall10 wrote: post #27
rohaga wrote: post #3

Earlier quoted context omitted.

> So this only switches models if the cache is cold Not exactly. It can also make sense if the cache is hot to switch models. For example, we might currently be on Deepseek and the next task is SO HARD that it doesn't make sense to not use frontier (1 turn to crack versus 100 turns for Deepseek to crack). Likewise, the router may judge that it's likely that the next few set of tool-calls will be ridiculously easy, an…

Seems like it might be more advantageous to just adjust reasoning effort to retain cache. Maybe in some exceptional cases where there will be a ton more inference to solve the problem, but going significantly dumber in that case seems counterintuitive. I can really only see the utility of things like spawning subagents to a lower tier model from another provider, and that's something harnesses can already handle (ie.…

I agree that adjusting the reasoning effort to retain cache is a huge thing! But even doing that automatically is currently a challenge for people to figure out and do well, and costs mental energy when perhaps it doesn't need to. For example, there is GPT-5.6-Sol low, med, high, xhigh, max, and lots of "rules of thumb" that people develop on which one to use when.

Re: Launch HN: Tokenless (YC S26) – Automatic model switching to save money

#34

Post by rohaga »

RuiWang0811 wrote: post #21

Does this mean you have to retrain routing rules every time a new model gets released? I imagine since the price/token (or rather the amount of work that can be done per token) does not monotonically increase with new models, that the routing logic has to change all the time

Yeah this is a great observation--we have to collect some more data to understand new models as they come out. However, by having this composable architecture that separates "confidence prediction" and the actual "routing layer," this makes it a lot easier to add new models/restrict the models rather than re-training from scratch.

Re: Launch HN: Tokenless (YC S26) – Automatic model switching to save money

#35

Post by rohaga »

popPopBoom wrote: post #25

Interesting approach. The multi-model progress monitoring idea is clever, most routing I've seen is either static rules or a cheap classifier that picks once upfront. Querying in parallel and deciding mid-turn feels different. One thing I'm curious about: how do you handle the latency hit from spinning up multiple models on the harder turns? Does the user-facing latency still feel competitive with just going straight…

Since we fan out in parallel, by definition the latency is only at worst the latency of the slowest model. The model in between is fairly light, so it seems to not materially affect things. Deciding mid-turn also lets us resolve earlier, often saving on the latency question. We hope to publish a detailed study about this soon.

We are exposing the routing decision information already, scroll over the response in the Dashboard and it'll tell you what models it considered and what it chose.

Re: Launch HN: Tokenless (YC S26) – Automatic model switching to save money

#36

Post by seizethecheese »

Super interesting approach. It's probably novel. I can say this because I've been working on something similar (while building a code version of http://pellmell.ai).

I'm skeptical though. In order to pick which model is on the right trajectory, you actually need intelligence. But real intelligence would make your system painfully slow and more expensive. I suspect you're using a classifier of some sort, but I also suspect what it's really measuring is confidence.

Most likely, this is a fantastic approach for the kind of problem where there's uncertainty but only one correct solution. But this is going to be really bad for cases where there are many potential solutions, some of which look good but are in fact bad. You only show one benchmark, and I'm wondering if it happens to be nicely shaped for this kind of router. Have you run it on DeepSWE?

Re: Launch HN: Tokenless (YC S26) – Automatic model switching to save money

#37

Post by rohaga »

seizethecheese wrote: post #36

Super interesting approach. It's probably novel. I can say this because I've been working on something similar (while building a code version of http://pellmell.ai ). I'm skeptical though. In order to pick which model is on the right trajectory, you actually need intelligence . But real intelligence would make your system painfully slow and more expensive. I suspect you're using a classifier of some sort, but I also…

We have run it on DeepSWE, check out our blog post for details of how we leverage the intelligence of the smart models https://usetokenless.com/blog/building-tokenless/

Re: Launch HN: Tokenless (YC S26) – Automatic model switching to save money

#38

Post by seizethecheese »

rohaga wrote: post #37
seizethecheese wrote: post #36

Super interesting approach. It's probably novel. I can say this because I've been working on something similar (while building a code version of http://pellmell.ai ). I'm skeptical though. In order to pick which model is on the right trajectory, you actually need intelligence . But real intelligence would make your system painfully slow and more expensive. I suspect you're using a classifier of some sort, but I also…

We have run it on DeepSWE, check out our blog post for details of how we leverage the intelligence of the smart models https://usetokenless.com/blog/building-tokenless/

Ah I didn’t see the tabs in the benchmark chart. Performance is worse on the two coding benchmarks and cheaper which makes sense.

I did read the blog post and I’m not sure what you mean by your reply. I don’t think the blog post addresses my comment.

Re: Launch HN: Tokenless (YC S26) – Automatic model switching to save money

#40

Post by hmokiguess »

I don't understand how you can make a business here. It seems to me you would end up optimizing your router for bottom line at some point and quality would degrade.

I worry about overfitting and bias in the router algorithm. Sometimes temperature and or variance that doesn't align with a directive leads to a faster task resolution, which may seem higher cost but overall task cost ending up cheaper.

Post reply on HN