Excited to play with this.
Let me know if you have any questions!
Show HN: Optimize and serve models with Fable quality at half the cost
21–30 of 34 posts
Re: Show HN: Optimize and serve models with Fable quality at half the cost
#22The title is misleading. This is model routing, not distillation.
Fixed formatting which will help with readability. We do routing, distillation, and token compaction.
Re: Show HN: Optimize and serve models with Fable quality at half the cost
#23[flagged]
Re: Show HN: Optimize and serve models with Fable quality at half the cost
#24Interesting approach. What does the cold-start phase look like for a new agent? How many traces or runs do you typically need before the router has enough signal to safely offload tasks from the frontier model??
Re: Show HN: Optimize and serve models with Fable quality at half the cost
#25Not sure I get it. The model you're improving is local? If so how do you even calculate cost compared to an API
Open source models. wmo routes requests between frontier models and open source models that continuously train using Tinker. As the smaller models improve, more traffic gets routed to them. Calculating cost is just tokens in/out.
What are the costs to train and use the Tinker models?
Re: Show HN: Optimize and serve models with Fable quality at half the cost
#26Interesting approach. What does the cold-start phase look like for a new agent? How many traces or runs do you typically need before the router has enough signal to safely offload tasks from the frontier model??
Technically 0 because
a) it ingests your already existing traces and does an initial training run
b) in the app we'll have pre-trained routers you can start with that will then learn over time
Re: Show HN: Optimize and serve models with Fable quality at half the cost
#27Earlier quoted context omitted.
Open source models. wmo routes requests between frontier models and open source models that continuously train using Tinker. As the smaller models improve, more traffic gets routed to them. Calculating cost is just tokens in/out.
What are the costs to train and use the Tinker models?
Expensive, in the thousands. We have our own infra in house and are working on bringing these costs down
Re: Show HN: Optimize and serve models with Fable quality at half the cost
#28Earlier quoted context omitted.
What are the costs to train and use the Tinker models?
Expensive, in the thousands. We have our own infra in house and are working on bringing these costs down
But in the thousands can pencil out if you're saying tens of thousands over calling the frontier models, no?
Re: Show HN: Optimize and serve models with Fable quality at half the cost
#29Re: Show HN: Optimize and serve models with Fable quality at half the cost
#30Probably a silly question, but ...: if it's optimizing itself purely inside a simulator, what keeps it from just exploiting the simulator's glitches to look good on paper, while not being so good when it hits the real world?