Congrats on the launch. To me this feels like DVC but with a slightly more convenient python API, and without the pipeline which might be really great. How are you organizing experiments though? Without Git it seems too easy to have experiments become unusable. And filtering according to accuracy is not really a solution..
Launch HN: Replicate (YC W20) – Version control for machine learning
41–50 of 57 posts
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#42Hello HN! We're Ben & Andreas, and we made Replicate. It's a lightweight open-source tool for tracking and analyzing your machine learning experiments: https://replicate.ai/ Andreas used to do machine learning at Spotify. He built a lot of ML infrastructure there (versioning, training, deployment, etc). I used to be product manager for Docker's open source projects, and created Docker Compose. We built https://www.ar…
Just commenting to say how awesome arxiv-vanity is. I didn’t know it existed, but I’ve wanted it to exist for years! Thanks!
It isn’t able to render all the papers i tried but still very useful.
Add a dark theme to it and I’ll be forever happy!
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#43Congrats on the launch! Very exciting tool. I'm one of the creators of https://DAGsHub.com which lets you host and collaborate on data science projects – think data science pull requests and data merging. We're integrated with DVC for data versioning and experiments, but what you're building is definitely super interesting.
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#44Earlier quoted context omitted.
Wait? So you can upload to Amazon or Google but nowhere else? Like to your own servers, for example?
You can save data to a path on the filesystem, so one way to do this is with a network mount. Lots of academic departments have their own GPU clusters, and they tend to have a shared network filesystem. We want to have more ways to do this though. We were close to adding SFTP support, but didn't get round to it. Another method could be to implement our own server, but we're trying to keep it simple for now. I'd be cu…
If you support S3 compatible storage providers that could mean people can use things like minio, that could be a shorter path to supporting lots of backends.
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#45What is the best way to do dataset versioning when using Replicate? I get that Replicate saves the dataset version but not the data itself. Is DVC a good fit?
Second question - can you tag or attach a comment to a group of experiments?
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#46How does this compare to tools like neptune.ai, weights and biases and so on? I can see the advantage of having control of one's data, whereas these tools use their own servers. However what I love about them is the amazing UI that allows me to compare experiments.
This came out of a practical problem: at Spotify, Andreas couldn't let any data leave their network. He wasn't going to go through procurement to buy an enterprise version of one of those products, so his only option left was open source software. But it's also out of principle: we think such a foundational thing needs to be open source. There is a reason most people use Git and not Perforce. Replicate can work along…
The idea looks very good!
I have to say however that I am a bit wary of notebooks. I agree how great they are to visualize things, but they also tend to have a negative effect on the coding standards of a team. This is again probably a personal bias and they look to be suited for this use case.
Great launch anyway!
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#47What is the best way to do dataset versioning when using Replicate? I get that Replicate saves the dataset version but not the data itself. Is DVC a good fit? Second question - can you tag or attach a comment to a group of experiments?
Great questions! At the moment we recommend passing dataset URIs as params to replicate.init(): https://replicate.ai/docs/guides/training-data, but of course this assumes immutable and stable URIs.
DVC would definitely be a good fit, and we have a ticket on our roadmap to integrate Replicate with DVC, Tecton, etc. https://github.com/replicate/replicate/issues/294
We also have a roadmap ticket for grouping experiments: https://github.com/replicate/replicate/issues/297, but for now we're recommending params for tags as well.
If you have ideas for the design of these features, we really appreciate feedback and comments on these Github issues!
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#48nit: "Throw away your spreadsheet" scares me a little. I love spreadsheets, and think there are 100x+ more users of spreadsheets than notebooks (though the overlap of notebook users and ML users is probably close to 1, so I see your point). I would always save my experiment results so they were ready to analyze in spreadsheets (and other vis tools).
Do you not think that using spreadsheets for ML experiment tracking is a symptom of broken tooling? I'm asking because one of the reasons we're building our platform[0] with automatic experiment tracking, collaborative notebooks, and a bunch of things, is because experiment tracking was inconsistent between team members.
Differences in tools used: (spreadsheets, flat files, logs, pen and paper, human memory). Forgetting to do it. Snippets to do it flying around. Different locations (laptop, group workstation, git repository, cloud sheet). Dissociated from the notebook that produced the model.
Tighter tracking should answer questions like: what notebook ran on which data and produced which model with which parameters and which scores? Then questions like: give me all notebooks that ran on this dataset which produced a model with scores that are [condition].
Once you do that, the "spreadsheet" can just be a "view" of the underlying data. Something you can export as, but not the thing itself.
I think it's good there are tools with this granularity that can be composed.
- [0]: https://iko.ai
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#49Earlier quoted context omitted.
Yeah, I agree this space is crowded. But we’ve found so few ML researchers/engineers are actually using these tools. This could either be that people aren’t aware of them yet, or that they’re not good enough. I think it’s a mix of both, honestly, but we’re betting that there’s more of the latter in the mix. :) I could do comparisons of each of these tools, and some of them are solving quite different problems, but th…
Can you share any of the data or market research on this? I am an ML manager in a large ecommerce firm and we stand up our own feature store, experiment tracking system and model training diagnostic / metric system (a la Tensorboard). It is exceedingly easy to DIY, I’ve been doing that stuff DIY with teams of less than 8 engineers for many years. What I’ve seen is that most vendor solutions aren’t flexible enough. Ma…
I am in a similar role and have the exact same opinion. Most vendors lack flexibility, can have significant costs even for small teams, and, at the end of the day, an in-house built solution isn't that difficult.
I think it's going to be a tough road to build a sustainable business in this space.
Re: Launch HN: Replicate (YC W20) – Version control for machine learning
#50Great to see this! I've been looking at alternatives for experiment tracking, but most of what I've come across have looked way too heavy (and expensive) for a small team.
Is R support planned? Or might this just work already with reticulate?