Viewing profile — danlenton

[flagged]

I think the main benefit is improved speed and parallelism. Very similar to https://thinkingmachines.ai/blog/interaction-models/

We just initialize a random latent vector for each model, and then jointly train each of these unique latent vectors :)

Currently the motivation is mainly speed. For the really easy ones like "hey, how's it going?" or "sorry I didn't hear you, can you repeat?" you can easily send to Llama3 etc. Ofc …

Interesting, do you have any hunch as to why this is? We've seen in more verticalized apps where the underlying model is hidden from the user (sales call agent, autopilot tool, sup…

no down side

One use case is optimizing agentic systems, where a custom router [ https://youtu.be/9JYqNbIEac0 ] is trained end-to-end on the final task (rather than GPT4-as-a-judge). Both the i…

Thanks for sharing, will get this fixed now!

If you do test it out, feel free to ping me with any questions!

Makes sense, however I would clarify that we don't need to make the final decision. If you're using the neural scoring function as an API, then you can just get predictions about h…

Thanks for weighing in. I'm sure for your setup right now, our router in it's current form would not be useful for you. This is the very first version, and the scope is therefore r…

duly noted!

however, as janekm says, we can't charge just based on cost savings. We would need the router points to be sufficiently compelling wrt quality, speed and cost (including our own ma…

Yeah that's a great point, something we'll keep in mind as we work out the final business model. Thanks!

Thanks! Ipsos is also a great analogous example, I hadn't thought of that.

Makes sense, thanks a lot for the feedback. We're pretty confident that future versions of our router will provide sufficient value where we can take margins here, we therefore don…

It's on the roadmap! Hopefully will be added next week

Yes the benchmarks are ongoing, we continually plot the speed and cost across time in our runtime benchmarks [ https://unify.ai/benchmarks ], and we use this live data when plotted…

Currently, we simply use GPT4-as-a-judge, with a general system prompt we've written which is task agnostic. This is then used to train the neural scoring function, which predicts …

Sure! Basically traditional MoE has several linear layers, and the network learns to route down those paths, based on the training loss (similar to how CNNs learn through max-pooli…

So the neural scoring introduces ~20ms latency, but this only impacts time-to-first-token (not inter-token-latency). When using our public endpoints there is an additional ~150ms l…

lol well sure, that too