CometML wants to do for machine learning what GitHub did for code
1–10 of 65 posts
Re: CometML wants to do for machine learning what GitHub did for code
#2Hi, I’m one of the founders of Comet.ml. We built comet.ml to allow machine learning teams to automatically track their machine learning code, experiments, hyperparameters and results. We think that reproducibility is really important so we’re also giving free access to students, academics and open source projects.
Feedback is welcome. Ask me anything.
Re: CometML wants to do for machine learning what GitHub did for code
#3When I saw that name the first thing I thought about was some new SML compiler or like MoscowML.
Re: CometML wants to do for machine learning what GitHub did for code
#4Hi, I’m one of the founders of Comet.ml. We built comet.ml to allow machine learning teams to automatically track their machine learning code, experiments, hyperparameters and results. We think that reproducibility is really important so we’re also giving free access to students, academics and open source projects. Feedback is welcome. Ask me anything.
[deleted]
Re: CometML wants to do for machine learning what GitHub did for code
#5When I saw that name the first thing I thought about was some new SML compiler or like MoscowML.
[deleted]
Re: CometML wants to do for machine learning what GitHub did for code
#6[deleted]
Re: CometML wants to do for machine learning what GitHub did for code
#7Great to see ML "governance" work being done on the training-part of the pipeline. Seems like this provides a Domino Data Labs based dashboards but without the walled garden environment.
I've yet to see similar great initiatives also tackling the deployment-part. E.g. something similar you can stick on top of your model's API (or scheduled batch predictive outputs), as well as incoming instances, to monitor usage patterns, population shifts through time, probability distributions, newly popping up missing values or categorical levels, logs, etc, in order to provide warning lights to indicate that a retraining might be in order, for instance.
Google's "What's your ML test score" paper provides some great insights, but I hope someone will tackle this with a turnkey solution as well.
Re: CometML wants to do for machine learning what GitHub did for code
#8Can you talk about how CometML fits into the real world state of ML training tracking, which is a pretty terrible Wild West of non-reproducible practices and processes?
There have been articles and comments here on HN about the sorry state of ML trackability, with papers being published on models whose training is not reproducible because no one really knows how it was trained. One in particular (I apologize for not having retained the link) described researchers starting with partially trained models they had lying around (with undocumented and unknown prior training applied), manually changing hyper parameters mid training while watching the learning progress, swapping different training sets in and out, and etc.
From what I see, the problems in ML reproducibility aren’t in the code, they are in the external human processes that are used to drive and train the models (essentially bad DevOps practices more than bad dev practices). Do you help with these kind of real-world trackability and reproducabilty scenarios?
Re: CometML wants to do for machine learning what GitHub did for code
#9I get so excited when I see ___ML but then I'm let down when I realize the context is Machine Learning instead of Meta Language. Not that Machine Learning isn't cool, but I really like ML family languages.
Re: CometML wants to do for machine learning what GitHub did for code
#10Hi, I’m one of the founders of Comet.ml. We built comet.ml to allow machine learning teams to automatically track their machine learning code, experiments, hyperparameters and results. We think that reproducibility is really important so we’re also giving free access to students, academics and open source projects. Feedback is welcome. Ask me anything.
Are you planning to open source it?
A lot of your competitors have, like http://pipeline.ai/, https://github.com/pachyderm/pachyderm and recently https://github.com/polyaxon/polyaxon.