Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CometML wants to do for machine learning what GitHub did for code

techcrunch.com

Post reply on HN
Print view View original

51–60 of 65 posts

Re: CometML wants to do for machine learning what GitHub did for code

#51

Post by nicodjimenez »

henripal wrote: post #50
nicodjimenez wrote: post #25

That's cool! I also wrote my own service https://losswise.com

Cool. I'm guessing from trying it out that you're using Highcharts. I've run into really unpleasant memory leaks/slowness when streaming data (especially when the existing chart is already thousands of data points). Are you seeing something similar?

Yes using Highcharts. You've had issues with Highcharts? Yeah it's not designed to stream data extremely rapidly but it's a great "good enough" product, especially for something like Losswise where the differentiation is the overall design and architecture and developer experience, not the prettiest possible graphs.

Re: CometML wants to do for machine learning what GitHub did for code

#52

Post by henripal »

nicodjimenez wrote: post #51
henripal wrote: post #50

Earlier quoted context omitted.

Cool. I'm guessing from trying it out that you're using Highcharts. I've run into really unpleasant memory leaks/slowness when streaming data (especially when the existing chart is already thousands of data points). Are you seeing something similar?

Yes using Highcharts. You've had issues with Highcharts? Yeah it's not designed to stream data extremely rapidly but it's a great "good enough" product, especially for something like Losswise where the differentiation is the overall design and architecture and developer experience, not the prettiest possible graphs.

Yes... For example if I'm running three experiments at the same time, auto-refreshing the chart every two seconds, it essentially freezes the app to a crawl after a thousand points or so. So we reverted to manual updates.

If you know of any better alternatives for data streaming, I'm curious. I tried benchmarking a couple libs recently: https://github.com/henripal/ChartingLibBenchmark

Re: CometML wants to do for machine learning what GitHub did for code

#53

Post by andbberger »

sbraden wrote: post #34

Since a few people commented on the cost of using CometML (or its competitors), I wanted to suggest an open source project (that's been around for a while) I found helpful for organizing ML experiments and tracking. It has two different frontends to choose from (I like SacredBoard). If you like open source this might be the ML experiment tracker for you! edit (forgot the link): https://github.com/IDSIA/sacred

+1 for sacred. It's changed my life.

Re: CometML wants to do for machine learning what GitHub did for code

#55

Post by nicodjimenez »

henripal wrote: post #52
nicodjimenez wrote: post #51

Earlier quoted context omitted.

Yes using Highcharts. You've had issues with Highcharts? Yeah it's not designed to stream data extremely rapidly but it's a great "good enough" product, especially for something like Losswise where the differentiation is the overall design and architecture and developer experience, not the prettiest possible graphs.

Yes... For example if I'm running three experiments at the same time, auto-refreshing the chart every two seconds, it essentially freezes the app to a crawl after a thousand points or so. So we reverted to manual updates. If you know of any better alternatives for data streaming, I'm curious. I tried benchmarking a couple libs recently: https://github.com/henripal/ChartingLibBenchmark

Yeah I have no idea. If you need really high performance that Highcharts doesn't provide you probably need to write your own specialized charting library.

Re: CometML wants to do for machine learning what GitHub did for code

#56

Post by mmq »

ah- wrote: post #10
gidim wrote: post #2

Hi, I’m one of the founders of Comet.ml. We built comet.ml to allow machine learning teams to automatically track their machine learning code, experiments, hyperparameters and results. We think that reproducibility is really important so we’re also giving free access to students, academics and open source projects. Feedback is welcome. Ask me anything.

Are you planning to open source it? A lot of your competitors have, like http://pipeline.ai/ , https://github.com/pachyderm/pachyderm and recently https://github.com/polyaxon/polyaxon .

@ah- thanks for mentioning Polyaxon and congrats to the CometMl team for building this nice tool, it's good to see that many projects are trying to solve problems related to reproducibility in ML/DL, many people had to build an internal tool for the companies they work for to solve this issue, and many got frustrated after joining a new team and were not able to reproduce any results.

I would like to outline a couple of differences between CometML and Polyaxon, as mentioned before, we are also trying to solve issues related to technical debt in ML, but not only, Polyaxon tries also to simplify training and scheduling parallel and distributed learning. there are also a couple of differences, I see CometML as dashboard, Polyaxon does not have an extensive dashboard as CometML, but it leverages Tensorboard for most of the visualisations. We use the CLI or the API for programatic access to the platform. Most importantly, Polyaxon aims to be an open source and to be installed on premise or in the cloud, it solves the issue related to code tracking based on an internal git and a docker registry, and as someone else mentioned that resources for running an experiment could be an issue for future reproducibility, Polyaxon restarts the experiments with the same resources and dockerfiles, it also tracks hyper params as part of the configuration.

For hyper params tuning and suggestion, Polyaxon can also do hyper params search based on a couple of algorithms, and for the next release, it will include also a service similar to vizier for suggesting more experiments/group of experiments based on a given search space.

Disclaimer: I am the author of Polyaxon

Re: CometML wants to do for machine learning what GitHub did for code

#57

Post by zitterbewegung »

ellisv wrote: post #38

Data science has 3 areas for "versioning": 1. code versioning 2. data versioning 3. model versioning Code versioning is primarily dominated by GitHub and is a fairly saturated space (Bitbucket, GitLab). Data versioning is either not happening, or being done through regular data pulls, database snapshots, etc. It is not well standardized or adopted. CometML is tackling model versioning. It would be really nice to have…

This software performs model versioning. https://mitdbg.github.io/modeldb/

Re: CometML wants to do for machine learning what GitHub did for code

#58

Post by sandGorgon »

gidim wrote: post #39
ellisv wrote: post #38

Data science has 3 areas for "versioning": 1. code versioning 2. data versioning 3. model versioning Code versioning is primarily dominated by GitHub and is a fairly saturated space (Bitbucket, GitLab). Data versioning is either not happening, or being done through regular data pulls, database snapshots, etc. It is not well standardized or adopted. CometML is tackling model versioning. It would be really nice to have…

We actually do code and model versioning (and simple data versioning). One thing to keep in mind is that code/results/hyperparams must be coupled. If you have a git branch with some training code and you do not know what the hyperparams/results are then it's not very valuable.

Could you talk more about a simple data versioning architecture. I have wondered about this coupling problem and would love to hear more.

Re: CometML wants to do for machine learning what GitHub did for code

#59

Post by nicodjimenez »

ellisv wrote: post #38

Data science has 3 areas for "versioning": 1. code versioning 2. data versioning 3. model versioning Code versioning is primarily dominated by GitHub and is a fairly saturated space (Bitbucket, GitLab). Data versioning is either not happening, or being done through regular data pulls, database snapshots, etc. It is not well standardized or adopted. CometML is tackling model versioning. It would be really nice to have…

1. and 3. can be combined with git lfs

Re: CometML wants to do for machine learning what GitHub did for code

#60

Post by gidim »

sandGorgon wrote: post #58
gidim wrote: post #39

Earlier quoted context omitted.

We actually do code and model versioning (and simple data versioning). One thing to keep in mind is that code/results/hyperparams must be coupled. If you have a git branch with some training code and you do not know what the hyperparams/results are then it's not very valuable.

Could you talk more about a simple data versioning architecture. I have wondered about this coupling problem and would love to hear more.

I was mostly referring to coupling code with results. For example you have a code that loads a dataset from S3 and then trains a neural network. If you only use git you're likely to lose the hyperparms info (which is often passed as command line arguments) and your metrics/results.
Post reply on HN