CometML wants to do for machine learning what GitHub did for code • Hacker News Board

Re: CometML wants to do for machine learning what GitHub did for code

#61

Post by ReverseCold » Thu, Apr 05, 2018, 9:11 PM UTC

gidim wrote: post #2

Hi, I’m one of the founders of Comet.ml. We built comet.ml to allow machine learning teams to automatically track their machine learning code, experiments, hyperparameters and results. We think that reproducibility is really important so we’re also giving free access to students, academics and open source projects. Feedback is welcome. Ask me anything.

Does "did what GitHub did for code" sound like a negative thing to anyone else?

It turned a decentralized platform (git) into basically the only place individuals store code.

Re: CometML wants to do for machine learning what GitHub did for code

#62

Post by davvolun » Thu, Apr 05, 2018, 9:42 PM UTC

ellisv wrote: post #38

Data science has 3 areas for "versioning": 1. code versioning 2. data versioning 3. model versioning Code versioning is primarily dominated by GitHub and is a fairly saturated space (Bitbucket, GitLab). Data versioning is either not happening, or being done through regular data pulls, database snapshots, etc. It is not well standardized or adopted. CometML is tackling model versioning. It would be really nice to have…

Interesting. Can you point me to a deeper discussion of this division of "versioning"?

I'm inclined to think something like Django data migrations or EntityFramework Code First Migrations tackles what I immediately thought of as "model versioning" and to some degree "data versioning" (though incomplete or probably impossible, for some things).

Re: CometML wants to do for machine learning what GitHub did for code

#63

Post by jorgemf » Fri, Apr 06, 2018, 4:25 AM UTC

shackenberg wrote: post #21

jorgemf wrote: post #17

I would like to use it but I think the price doesn't justify the tool. For a team of 5 people github is $25 a month, you are $745 a month. I do understand a price a bit higher that github but not 30 times more expensive.

I like, that the price is so high. That makes it a least seem to be a sustainable business. And when you pay your ML people 10k a month, the $25 is less than the coffee they will drink in the office.

145 per user per month, not 25. Not all countries pay 10k a month to their employees

Re: CometML wants to do for machine learning what GitHub did for code

#64

Post by jorgemf » Fri, Apr 06, 2018, 4:29 AM UTC

gidim wrote: post #20

jorgemf wrote: post #17

I would like to use it but I think the price doesn't justify the tool. For a team of 5 people github is $25 a month, you are $745 a month. I do understand a price a bit higher that github but not 30 times more expensive.

Thanks @jorgemf. Keep in mind that $745 also includes unlimited usage of our hyper-parameter optimization service.

That service doesn't justify the price tag for me.

Re: CometML wants to do for machine learning what GitHub did for code

#65

Post by TorsteinHonsi » Fri, Apr 06, 2018, 8:22 AM UTC

henripal wrote: post #52

nicodjimenez wrote: post #51

Earlier quoted context omitted.

Yes using Highcharts. You've had issues with Highcharts? Yeah it's not designed to stream data extremely rapidly but it's a great "good enough" product, especially for something like Losswise where the differentiation is the overall design and architecture and developer experience, not the prettiest possible graphs.

Yes... For example if I'm running three experiments at the same time, auto-refreshing the chart every two seconds, it essentially freezes the app to a crawl after a thousand points or so. So we reverted to manual updates. If you know of any better alternatives for data streaming, I'm curious. I tried benchmarking a couple libs recently: https://github.com/henripal/ChartingLibBenchmark

As a Highcharts developer, I had a look at your benchmarking, and have some thoughts about optimizing for Highcharts. The first step is to turn off animation, which helps a lot. The default Highcharts animation on addPoint is 250ms, so with a refresh rate of 100ms you will get a lot of redrawing going on for nothing. The second thing that possibly optimizes a bit is to use hard-coded axis values so that it doesn't have to recompute axis values for each iteration.

With those modifications the performance is much better: http://jsfiddle.net/highcharts/1o5ghqc8/