How do you compare to a platform like Arize? https://arize.com/
Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models
11–20 of 34 posts
Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models
#12How do you compare to a platform like Arize? https://arize.com/
Being open-source is a key differentiator. With us being open-source, UpTrain can be easily customized for any specific use-case.
With UpTrain, one can define custom measures to monitor upon, add custom algorithms for model stability or drift detection as well as fill in any integration gaps in terms of using us in production
Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models
#13Congrats on the launch, looks like a cool product! Just scanned the docs, so not super sure if my specific use case is supported. I previously worked on a content recommendation system for academic users. We often wanted to go back and look through specific user sessions to see if the recommendations made sense in the context of their activity. So, ground truth data was kind of available, but only at a later time. Is…
Interesting thought. We ran into the same issue working on a recommendation system engine as well and previously tried to build a solution around it. Curious to know what’s driving your interest in post-analysis?
Oh, to be specific, the platform was oriented towards test practice and our objective was to recommend questions in a sequence.
One good strategy that correlated with session length for us was asking questions that were neither too difficult or easy, based on what we knew of the user's level at that instant. The post hoc analysis was really meant to dig into multiple user sessions and see if the current method was working and evaluate counterfactual strategies.
I imagine the Uptrain product could help us segment user cohorts, find out which ones aren't performing super well etc. Would love to hear what you ended up building too?
Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models
#14How does your model illustrate specific feature combination for deep learning models - let's say instance segmentation
There are two ways: 1. We use model-inferred embeddings. Say, for the instance segmentation task, we use deep learning networks to transform the input image into a dense embeddings representation, on top of which we run clustering and density estimation to find if the given embedding/image/feature combination is an outlier (or belongs to low-density region) 2. We allow users to define custom signals to identify edge-…
[deleted]
Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models
#15Congrats on the launch, looks like a cool product! Just scanned the docs, so not super sure if my specific use case is supported. I previously worked on a content recommendation system for academic users. We often wanted to go back and look through specific user sessions to see if the recommendations made sense in the context of their activity. So, ground truth data was kind of available, but only at a later time. Is…
Interesting thought. We ran into the same issue working on a recommendation system engine as well and previously tried to build a solution around it. Curious to know what’s driving your interest in post-analysis?
Also, happy to share our learnings of working with a social media customer. For them, key motivation was to understand where the model is failing and hence, understand how to improve it. They started with offline experiments with focus on improving AUC but the graph saturates pretty quickly. While an incremental model improvement didn't lead to a sizeable change in offline metrics, it can impact retention of a certain user group and hence, overall revenue. They are using us to get insights on online experimentation. They have defined custom measures to monitor the distribution of model outputs and are able to determine the difference in model's performance much more effectively as compared to relying on changes in business metrics like retention, revenue, etc. This helps them to find out poor-performing cohorts and roll-out model improvements in weeks, not months. Would also love to hear what issues you ran into?
Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models
#16How do you compare to a platform like Arize? https://arize.com/
Being open-source is a key differentiator. With us being open-source, UpTrain can be easily customized for any specific use-case. With UpTrain, one can define custom measures to monitor upon, add custom algorithms for model stability or drift detection as well as fill in any integration gaps in terms of using us in production
Additionally, refinement is a key focus of ours. Figuring out the best data points to retrain the model upon has twin benefits:
1) It provides automated issue resolution and saves data scientists' effort to debug and fix their models. 2) It allows us to reduce false positives in alerting: we send alerts only when we see a dip in model performance, or retraining can lead to improved model accuracy.
Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models
#17Very cool product and best of all you made it open source to remove the barrier of installing it in someone’s own environment.
Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models
#18Very cool product and best of all you made it open source to remove the barrier of installing it in someone’s own environment.
Thanks a lot! Yes, we are big believers of open-source :)
Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models
#19Earlier quoted context omitted.
Being open-source is a key differentiator. With us being open-source, UpTrain can be easily customized for any specific use-case. With UpTrain, one can define custom measures to monitor upon, add custom algorithms for model stability or drift detection as well as fill in any integration gaps in terms of using us in production
Additionally, refinement is a key focus of ours. Figuring out the best data points to retrain the model upon has twin benefits: 1) It provides automated issue resolution and saves data scientists' effort to debug and fix their models. 2) It allows us to reduce false positives in alerting: we send alerts only when we see a dip in model performance, or retraining can lead to improved model accuracy.
Awesome! Big fan of OS -- arize is powerful yet expensive, so I think there's a big market there. Alerting is super tough to get right, and false positives are often worse than no alerting at all. In ML its even harder cause "data looks weird" is like 90% of the bugs.
Anyway, congrats! Excited to see where you go with this.
Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models
#20Excited to see more people building in this space. From what we've seen with customers it's critical to be able to compare what you're seeing in production to what you trained on (rather than historical period). That's almost the textbook definition of drift. Do you have a sense on how to approach that?
At Comet.com (disclaimer: i'm the CEO/Co-founder) we provide experiment tracking and artifacts management so we have the training distributions for comparison. I'm always curious how it looks like for a monitoring only solution