Launch HN: Replicate (YC W20) – Version control for machine learning

Hello HN!

We're Ben & Andreas, and we made Replicate. It's a lightweight open-source tool for tracking and analyzing your machine learning experiments: https://replicate.ai/

Andreas used to do machine learning at Spotify. He built a lot of ML infrastructure there (versioning, training, deployment, etc). I used to be product manager for Docker's open source projects, and created Docker Compose.

We built https://www.arxiv-vanity.com/ together for fun, which led to us teaming up to build more tools for ML.

We spent a year talking to lots of people in the ML community and building all sorts of prototypes, but we kept on coming back to a foundational problem: not many people in machine learning use version control.

This causes all sorts of problems: people are manually keeping track of things in spreadsheets, model weights are scattered on S3, and results can’t be reproduced.

So why isn’t everyone using Git? Git doesn’t work well with machine learning. It can’t store trained machine learning models, it can’t handle key/value metadata, and it’s not designed to record information automatically from a training script. There are some solutions for these things, but they feel like band-aids.

We came to the conclusion that we need a native version control system for ML. It’s sufficiently different to normal software that we can’t just put band-aids on Git.

We believe the tool should be small, easy to use, and extensible. We found people struggling to migrate to “AI Platforms”. A tool should do one thing well and combine with other tools to produce the system you need.

Finally, we also believe it should be open source. There are a number of proprietary solutions, but something so foundational needs to be built by and for the ML community.

Replicate is a first cut at something we think is useful: It is a Python library that uploads your files and metadata (like hyperparameters) to Amazon S3 or Google Cloud Storage. You can get back to any point in time using the command-line interface, analyze your results inside a notebook using the Python API, and load your models in production systems.

We’d love to hear your feedback, and hear your stories about how you’ve done this before.

Also – building a version control system is rather complex, and to make this a reality we need your help. Join us in Discord if you want to be involved in the early design and help build it: https://discord.gg/QmzJApGjyE

As someone who tried to use git to do this for large sets of data, I'm very glad this exists. Will be trying this out in the future.

Thanks!

DVC is closely tied to Git. We've heard people find that quite heavyweight when you're running experiments.

We think we can build a much better experience if we detach ourselves from Git. With Replicate, you just run your training script as usual, and it automatically tracks everything from within Python. You don't have to run any additional commands to track things.

DVC is really good for storing data sets though, and we see potential for integration there: https://github.com/replicate/replicate/issues/359

We talked to a bunch of MLflow users, and the general impression we got is that it is heavyweight and hard to set up. MLflow is an all-encompassing "ML platform". Which is fine if you need that, but we're trying to just do one thing well. (Imagine if Git called itself a "software platform".)

In terms of features, Replicate points directly at an S3 bucket (so you don't have to run a server and Postgres DB), it saves your training code (for reproducibility and to commit to Git after the fact), and it has a nice API for reading and analyzing your experiments in a notebook.

Congrats on the launch.

I have built an open source tool (called hyperML) for similar problem sometime back. I think the problem is not just storing in version control but being able to quickly retrieve those in live /test systems or containers.

Mounting and loading datasets and models is painful. It kind of what makes local training a better option.

If only the weights were version controlled by libraries (tf, pytorch or scikit) this whole problem will be much easier to solve.