Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models

#21

Post by sourabh03agr »

gidim wrote: post #20

Excited to see more people building in this space. From what we've seen with customers it's critical to be able to compare what you're seeing in production to what you trained on (rather than historical period). That's almost the textbook definition of drift. Do you have a sense on how to approach that? At Comet.com (disclaimer: i'm the CEO/Co-founder) we provide experiment tracking and artifacts management so we hav…

Completely agree! We have also seen our users more concerned about comparing the real-world distribution against the training data as compared to previous month's data (we found latter is more useful for PMs and setting alerts).

We currently allow users to specify their training data in the config which is used to initialise the UpTrain framework (in form of json file but are planning to support pytorch/tf data-loaders). In the background, the tool does all the binning and clustering to convert these continuous variables into discrete buckets to later calculate divergence, which is then used to quantify drift.

Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models

#22

Post by vvipgupta »

gidim wrote: post #20

Excited to see more people building in this space. From what we've seen with customers it's critical to be able to compare what you're seeing in production to what you trained on (rather than historical period). That's almost the textbook definition of drift. Do you have a sense on how to approach that? At Comet.com (disclaimer: i'm the CEO/Co-founder) we provide experiment tracking and artifacts management so we hav…

Thanks for the very relevant comment :) We provide users the option to attach their training data from csv/json (working to support loading from cloud storage provider or data lakes). We have illustrated this in some of our examples, such as the human orientation classification: https://github.com/uptrain-ai/uptrain/blob/main/examples/hum...

Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models

#24

Post by sourabh03agr »

shresh0202 wrote: post #23

I have seen a lot of observability solutions and they don't seem to work for Deep learning models. Can you explain why your approach will work, say from a language model perspective?

Yes, general observability tools don't work well for ML applications as they lack support for ML specific use cases such as attaching Ground Truth label, data drift, model bias, etc.

Beyond that, requirements for Deep learning models is even more nuanced. Say, for language models, we provide two key features to effectively monitor them:

1. We represent the text by an embedding (e.g. BERT) which are much more informative from a statistical distribution perspective to find out edge cases, low density regions, etc. Further we use Earth Moving Distance to quantify data drift in the multi-dimensional space.

2. We allow user-defined smart signals to be written on top of your model inputs/outputs. You can classify a certain prediction to be wrong if it doesn't follow grammar rules or have occurrence of certain keywords or the prediction is followed by a certain user behaviour pattern (if an user is not satisfied with the response of ChatGPT, it is expected that they will ask the same question again and again in different ways). All these customisations serve as a good proxy to actually observe the model's performance and find avenues to improve them.

Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models

#26

Post by vvipgupta »

elijahbenizzy wrote: post #19
vvipgupta wrote: post #16

Earlier quoted context omitted.

Additionally, refinement is a key focus of ours. Figuring out the best data points to retrain the model upon has twin benefits: 1) It provides automated issue resolution and saves data scientists' effort to debug and fix their models. 2) It allows us to reduce false positives in alerting: we send alerts only when we see a dip in model performance, or retraining can lead to improved model accuracy.

Awesome! Big fan of OS -- arize is powerful yet expensive, so I think there's a big market there. Alerting is super tough to get right, and false positives are often worse than no alerting at all. In ML its even harder cause "data looks weird" is like 90% of the bugs. Anyway, congrats! Excited to see where you go with this.

Thanks! Also, wondering how did you hear about Arize? Have you dealt with the pain of ML model monitoring in the past?

Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models

#27

Post by Areibman »

General question for the MLOps community: I usually see tools like this launch all the time. I'm eager to experiment, but I find my use cases never fit, and I end up just building a simple internal tool that does the job. Usually because the dataset or model is too unique.

Is it just me?

Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models

#28

Post by sourabh03agr »

Areibman wrote: post #27

General question for the MLOps community: I usually see tools like this launch all the time. I'm eager to experiment, but I find my use cases never fit, and I end up just building a simple internal tool that does the job. Usually because the dataset or model is too unique. Is it just me?

Can you describe your use case?

We also faced the same problem with other tools and hence building UpTrain with customisation at the core of it. Would be interesting to see if your use case fits

Re: Launch HN: UpTrain (YC W23) – Open-source performance monitoring for ML models

#29

Post by vvipgupta »

Areibman wrote: post #27

General question for the MLOps community: I usually see tools like this launch all the time. I'm eager to experiment, but I find my use cases never fit, and I end up just building a simple internal tool that does the job. Usually because the dataset or model is too unique. Is it just me?

Generally, MLOps helps in reducing engineering headaches. During our user interviews and customer calls, we realized very early that customization is key for ML model monitoring since all models are different. Thus, we have built the framework to lessen the engineering headache while allowing customizability (think PyTorch). Would love to know your thoughts on this.
Post reply on HN