Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CometML wants to do for machine learning what GitHub did for code

techcrunch.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: CometML wants to do for machine learning what GitHub did for code

#31

Post by XnoiVeX »

gidim wrote: post #20
jorgemf wrote: post #17

I would like to use it but I think the price doesn't justify the tool. For a team of 5 people github is $25 a month, you are $745 a month. I do understand a price a bit higher that github but not 30 times more expensive.

Thanks @jorgemf. Keep in mind that $745 also includes unlimited usage of our hyper-parameter optimization service.

Are there any similarities or distinctions between this article (link below) and how your system works to tune hyper-parameters?

https://blog.coast.ai/lets-evolve-a-neural-network-with-a-ge...

Re: CometML wants to do for machine learning what GitHub did for code

#32

Post by gidim »

XnoiVeX wrote: post #31
gidim wrote: post #20

Earlier quoted context omitted.

Thanks @jorgemf. Keep in mind that $745 also includes unlimited usage of our hyper-parameter optimization service.

Are there any similarities or distinctions between this article (link below) and how your system works to tune hyper-parameters? https://blog.coast.ai/lets-evolve-a-neural-network-with-a-ge...

This article seems to discuss genetic algorithms which could be used for hyperparam optimization. We use another method called Bayesian (GP) hyperparam optimization. According to our internal benchmarks and academic research Bayesian methods outperform genetic algorithms. Another thing to keep in mind that we automate the entire process for you. You only need to provide a list of parameters you'd like to tune.

Re: CometML wants to do for machine learning what GitHub did for code

#33

Post by henripal »

Cool, thanks.

For those of you who want to tinker, there's a much rougher, open source library based on Vuejs, postgres, and Flask with some momentum on GitHub right now, LabNotebook https://github.com/henripal/labnotebook

(Disclaimer: I'm one of the authors)

Re: CometML wants to do for machine learning what GitHub did for code

#34

Post by sbraden »

Since a few people commented on the cost of using CometML (or its competitors), I wanted to suggest an open source project (that's been around for a while) I found helpful for organizing ML experiments and tracking. It has two different frontends to choose from (I like SacredBoard). If you like open source this might be the ML experiment tracker for you!

edit (forgot the link): https://github.com/IDSIA/sacred

Re: CometML wants to do for machine learning what GitHub did for code

#35

Post by softawre »

sbraden wrote: post #34

Since a few people commented on the cost of using CometML (or its competitors), I wanted to suggest an open source project (that's been around for a while) I found helpful for organizing ML experiments and tracking. It has two different frontends to choose from (I like SacredBoard). If you like open source this might be the ML experiment tracker for you! edit (forgot the link): https://github.com/IDSIA/sacred

Did you forget to suggest the project? Is it "sacred"?

https://github.com/IDSIA/sacred

Re: CometML wants to do for machine learning what GitHub did for code

#36

Post by gidim »

p1esk wrote: post #29

How do you handle dataset/checkpoint management and versioning? Ideally with powerful dataset filtering options. Right now we’re using excel and panda dataframes, but are interested if your tool does it well.

Since we do not host your data we cannot provide filtering on the actual dataset content. We do allow you to track where the data was coming from and if it changed (by hash). Same for checkpoints, you can log their location (S3/local path) and hash.

Re: CometML wants to do for machine learning what GitHub did for code

#38

Post by ellisv »

Data science has 3 areas for "versioning":

1. code versioning 2. data versioning 3. model versioning

Code versioning is primarily dominated by GitHub and is a fairly saturated space (Bitbucket, GitLab). Data versioning is either not happening, or being done through regular data pulls, database snapshots, etc. It is not well standardized or adopted. CometML is tackling model versioning.

It would be really nice to have a single solution for all of these but that is unlikely. Hopefully new standards evolve from this.

Re: CometML wants to do for machine learning what GitHub did for code

#39

Post by gidim »

ellisv wrote: post #38

Data science has 3 areas for "versioning": 1. code versioning 2. data versioning 3. model versioning Code versioning is primarily dominated by GitHub and is a fairly saturated space (Bitbucket, GitLab). Data versioning is either not happening, or being done through regular data pulls, database snapshots, etc. It is not well standardized or adopted. CometML is tackling model versioning. It would be really nice to have…

We actually do code and model versioning (and simple data versioning). One thing to keep in mind is that code/results/hyperparams must be coupled. If you have a git branch with some training code and you do not know what the hyperparams/results are then it's not very valuable.

Re: CometML wants to do for machine learning what GitHub did for code

#40

Post by laythea »

yodon wrote: post #8

Can you talk about how CometML fits into the real world state of ML training tracking, which is a pretty terrible Wild West of non-reproducible practices and processes? There have been articles and comments here on HN about the sorry state of ML trackability, with papers being published on models whose training is not reproducible because no one really knows how it was trained. One in particular (I apologize for not…

Im no expert but reading that it sounds like a bunch of people poking at knobs and dials in order to see which dials work and which don't, and then not understanding why...?
Post reply on HN