Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CometML wants to do for machine learning what GitHub did for code

techcrunch.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: CometML wants to do for machine learning what GitHub did for code

#41

Post by jdoliner »

yodon wrote: post #8

Can you talk about how CometML fits into the real world state of ML training tracking, which is a pretty terrible Wild West of non-reproducible practices and processes? There have been articles and comments here on HN about the sorry state of ML trackability, with papers being published on models whose training is not reproducible because no one really knows how it was trained. One in particular (I apologize for not…

My sense is that CometML is tackling some but not all of the problems you're talking about here. Specifically I think it's pretty focused on tracking ML models so you can keep track of how models are performing, how they're improving etc. But I think, and the other comment seems to suggest, that external systems and practices will be required to make it so these results can be reproduced. Making ML experiments reproducible is a different and fairly challenging problem. You need to have a way to essentially snapshot everything that goes into training a model, training data, code, tuning parameters maybe even specific hardware it needs to run on. As you mention, this is as much a DevOps problem as it is an ML problem, maybe even more so. I think you'd have good luck pairing CometML with a system like Pachyderm [0] (which, full disclosure, is a system I helped write). Pachyderm can handle the Devopsy reproducibility piece and be paired with CometML to get insight into what's actually going on with the models.

[0] https://github.com/pachyderm

Re: CometML wants to do for machine learning what GitHub did for code

#42

Post by alexilliamson »

ellisv wrote: post #38

Data science has 3 areas for "versioning": 1. code versioning 2. data versioning 3. model versioning Code versioning is primarily dominated by GitHub and is a fairly saturated space (Bitbucket, GitLab). Data versioning is either not happening, or being done through regular data pulls, database snapshots, etc. It is not well standardized or adopted. CometML is tackling model versioning. It would be really nice to have…

Nice breakdown. I agree that data versioning is the one area with limited standardized options. I would add that in addition to versioning the data, there is also the related problem of integrating the 3 areas of versioning... tying the "data version" to the "model version" and the "code version". That seems to me like it might be a good place to start in tackling data versioning, or is that too trivial? Is there a product out there that already does this?

Re: CometML wants to do for machine learning what GitHub did for code

#43

Post by tedivm »

Their pricing is ridiculous. $50 per user per month for their cheapest plan that works with private spaces, $149 per user per month for self hosted.

If you're a company with 15 people working on a single model you're going to be paying more than a team with five people who have 20 different models they are working on. The actual load and cost for the service is completely detached from the price.

I was strongly considering looking at this for a project, but not at that cost.

Re: CometML wants to do for machine learning what GitHub did for code

#45

Post by tedivm »

suff wrote: post #44

Great idea if AWS had not already solved this with the release of SageMaker.

SageMaker is awesome but it's a very different product. There is nothing in SageMaker that lets you track performance over time and no fancy dashboard. That being said SageMaker's pricing is actually reasonable, and adding a dashboard on top of it shouldn't be that complicated.

Re: CometML wants to do for machine learning what GitHub did for code

#46

Post by jdoliner »

alexilliamson wrote: post #42
ellisv wrote: post #38

Data science has 3 areas for "versioning": 1. code versioning 2. data versioning 3. model versioning Code versioning is primarily dominated by GitHub and is a fairly saturated space (Bitbucket, GitLab). Data versioning is either not happening, or being done through regular data pulls, database snapshots, etc. It is not well standardized or adopted. CometML is tackling model versioning. It would be really nice to have…

Nice breakdown. I agree that data versioning is the one area with limited standardized options. I would add that in addition to versioning the data, there is also the related problem of integrating the 3 areas of versioning... tying the "data version" to the "model version" and the "code version". That seems to me like it might be a good place to start in tackling data versioning, or is that too trivial? Is there a p…

Pachyderm, a project I work on, is probably as close as you'll find to something that ties all 3 together. In my mind the major unsolved problem here was the data versioning so that's the first thing we tackled. Code versioning is already quite well solved so we just integrate with existing tools for that. I'm not convinced that model versioning is actually distinct from data versioning, models are just data after all. So I think without an established system for versioning models, such as Git + Github is for code, treating models as data and versioning them that way is good enough for government work. From what I can tell CometML isn't quite versioning models so much as tracking versions of models. It expects that models to be stored and versioned elsewhere but it gives you a way to get deeper insight into how those models are performing, how they're changing, the hyper-parameters used to train them etc. Tracking this is also a very important problem that CometML seems to solve quite elegantly.

Re: CometML wants to do for machine learning what GitHub did for code

#47

Post by nicodjimenez »

dspoka wrote: post #37
nicodjimenez wrote: post #25

That's cool! I also wrote my own service https://losswise.com

Do you have any information on pricing?

Current feature set is completely free as the service is in early beta stage. We plan to charge later for additional features and services. The project started as an internal project for Mathpix.com and therefore already pays for itself.

Re: CometML wants to do for machine learning what GitHub did for code

#49

Post by maksimum »

gidim wrote: post #20
jorgemf wrote: post #17

I would like to use it but I think the price doesn't justify the tool. For a team of 5 people github is $25 a month, you are $745 a month. I do understand a price a bit higher that github but not 30 times more expensive.

Thanks @jorgemf. Keep in mind that $745 also includes unlimited usage of our hyper-parameter optimization service.

Is hyper-parameter optimization a cherry on top or one of the core value propositions?

IME hyper-parameter optimization doesn't require much in terms of implementation effort (e.g. [1]), but requires compute. I would be surprised if a professional ML/DS user were to seriously consider paying for the implementation of the optimization.

[1] https://people.eecs.berkeley.edu/~kjamieson/hyperband.html

Re: CometML wants to do for machine learning what GitHub did for code

#50

Post by henripal »

nicodjimenez wrote: post #25

That's cool! I also wrote my own service https://losswise.com

Cool. I'm guessing from trying it out that you're using Highcharts. I've run into really unpleasant memory leaks/slowness when streaming data (especially when the existing chart is already thousands of data points). Are you seeing something similar?
Post reply on HN