nit: "Throw away your spreadsheet" scares me a little. I love spreadsheets, and think there are 100x+ more users of spreadsheets than notebooks (though the overlap of notebook users and ML users is probably close to 1, so I see your point). I would always save my experiment results so they were ready to analyze in spreadsheets (and other vis tools).
Launch HN: Replicate (YC W20) – Version control for machine learning
11–20 of 57 posts
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#12Hello HN! We're Ben & Andreas, and we made Replicate. It's a lightweight open-source tool for tracking and analyzing your machine learning experiments: https://replicate.ai/ Andreas used to do machine learning at Spotify. He built a lot of ML infrastructure there (versioning, training, deployment, etc). I used to be product manager for Docker's open source projects, and created Docker Compose. We built https://www.ar…
> it can’t handle key/value metadata What do you mean by that? Is a JSON no good? I guess you mean the diffs will be unordered?
Yep, and we can do lots of other nice things. We can produce nice tables with the key/value data, filter it ("show me all experiments with an accuracy greater than 0.9"), produce well-formatted diffs across an arbitrary number of things, give you a nice Python API for analyzing the data in a notebook, and so on.
There are some examples of these things on the home page, all of which would be very fiddly to do with JSON files in Git: https://replicate.ai/#features
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#13nit: "Throw away your spreadsheet" scares me a little. I love spreadsheets, and think there are 100x+ more users of spreadsheets than notebooks (though the overlap of notebook users and ML users is probably close to 1, so I see your point). I would always save my experiment results so they were ready to analyze in spreadsheets (and other vis tools).
Hi, Andreas here. Yes spreadsheets are great, and better than notebooks in many cases. But I always felt like I was doing something wrong when I used spreadsheets and markdown files to manually record metrics and hyperparameters for my experiments. It's error prone and easy to forget to update the spreadsheet with new experiments.
So we're trying to automate recording this metadata, but then give you that metadata in various ways for you to inspect it. One of those ways is actually spreadsheets: https://github.com/replicate/replicate/issues/289
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#14Earlier quoted context omitted.
Thanks! DVC is closely tied to Git. We've heard people find that quite heavyweight when you're running experiments. We think we can build a much better experience if we detach ourselves from Git. With Replicate, you just run your training script as usual, and it automatically tracks everything from within Python. You don't have to run any additional commands to track things. DVC is really good for storing data sets t…
Hey, one of the DVC maintainers here! TL;DR: I think it should be compared with the upcoming DVC feature - https://github.com/iterative/dvc/wiki/Experiments . Stay tuned - it'll be released very soon but you can try it now in beta. First of all, congrats on the launch! I do really like the aesthetics of the website, and the overall approach. It resonates with our vision and philosophy! Good feedback on experiments fe…
Would love to chat -- I'll shoot you an email. :)
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#15How does this compare to tools like neptune.ai, weights and biases and so on? I can see the advantage of having control of one's data, whereas these tools use their own servers.
However what I love about them is the amazing UI that allows me to compare experiments.
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#16Earlier quoted context omitted.
Hey, one of the DVC maintainers here! TL;DR: I think it should be compared with the upcoming DVC feature - https://github.com/iterative/dvc/wiki/Experiments . Stay tuned - it'll be released very soon but you can try it now in beta. First of all, congrats on the launch! I do really like the aesthetics of the website, and the overall approach. It resonates with our vision and philosophy! Good feedback on experiments fe…
Would love to chat -- I'll shoot you an email. :)
ghj
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#17How does this compare to tools like neptune.ai, weights and biases and so on? I can see the advantage of having control of one's data, whereas these tools use their own servers. However what I love about them is the amazing UI that allows me to compare experiments.
This came out of a practical problem: at Spotify, Andreas couldn't let any data leave their network. He wasn't going to go through procurement to buy an enterprise version of one of those products, so his only option left was open source software.
But it's also out of principle: we think such a foundational thing needs to be open source. There is a reason most people use Git and not Perforce.
Replicate can work alongside visualization tools -- your data is safe in your own S3 bucket, but you can use the hosted visualization tool to complement that.
You could also imagine visualization tools built on top of Replicate. One thing we've been thinking about is doing visualization inside notebooks. It's like a programmable Tensorboard: https://colab.research.google.com/drive/18sVRE4Zi484G2rBeOYj...
I'd be curious to hear your thoughts about that. It's pretty primitive so far, but we've got to start somewhere I suppose. :)
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#18Hello HN! We're Ben & Andreas, and we made Replicate. It's a lightweight open-source tool for tracking and analyzing your machine learning experiments: https://replicate.ai/ Andreas used to do machine learning at Spotify. He built a lot of ML infrastructure there (versioning, training, deployment, etc). I used to be product manager for Docker's open source projects, and created Docker Compose. We built https://www.ar…
Wait? So you can upload to Amazon or Google but nowhere else? Like to your own servers, for example?
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#19I was looking for something similar today. I just adopted it. :)
Thank you for your amazing work!
Do you have the intention to integrate it with PT Lightning or as a PT Lightning Logger?
It would be nice to have it maintained there.
It’s used all over huggingface-Transformers examples.
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#20Congrats on the launch.
Have you looked at https://comet.ml
If so, how do you compare to them?