Ask HN: What's Your CI/CD Workflow for Your Machine Learning Projects?

Full Disclosure (usually considered courteous to provide this yourself): @mikeyanderson is head of Marketing at Algorithmia.com.

It does look like a cool service, though!

Thanks for this; I really appreciate the detail here. There seems to be a lack of these kinds of explanations around.

One piece did give me a bit of surprise:

  You might also need to implement the inference code as well to get the speed you need

I've never had the super-low-latency requirements you have, but as you point out this seems amazingly error-prone. I'd love to hear anything else you can share about the cost-benefit analysis you do before deciding to go this route, and if there's any tools or languages you've had a better time with than others for this.

I don't work on computer vision specifically, and can generally suffer a few seconds of latency, but we have a two-stage process that I think applies fairly generally

1. all lowish-level production code (importing, transforming, modeling) is written in packages, which are thoroughly unit-tested with a CI system

2. that code is wrapped up into docker container(s) in separate repositories, which are built and integration-tested with CI. In addition to the Dockerfile and any testing scripts, there's usually a single code file here which handles I/O specifics, API endpoints, and primarily calls code from the package

This works well with R or Python and should work with others; we use Gitlab for the free private repos and awesome built-in CI.

This doesn't cover where the data or models are stored, but that varies more per-project for us. Lately we've been using Pachyderm and loving it, but you can get pretty far with a postgres instance for data and storing trained model objects in S3/GCS.

I have deployed ML algorithms into production including computer vision and data science models. From my experience, it's based on the application and cost we are comfortable with.

1. Keras/Tensorflow based algorithm(Applicable for any compute intensive or GPU-capable algorithm): Deployed the method (as a flask service) inside a Docker container along with a queueing system(for reliability w/ redis). We can now decide the server type and the kind of orchestration tool we can use to manage these containers. Following are some options for it,

    a. ECS on AWS
    b. Kubernetes 
    c. Docker Swarm
    d. custom orchestration tool

2. If your ML model is a simpler with a small (enough) model size, then using Lambdas on AWS would also work. This can provide high throughput and low cost per request if your computation time isn't very high.

Tips:

   i. Have the memory flush in the code after the service is used so that there is no memory leak. 
   ii. You can use tools such as htop to understand the memory usage.
   iii. Regarding system performance, you can use prometheus to gather stats along with grafana dashboard to view them.

I consider CI/CD essential for achieving a seamless workflow for a data scientist, and after having faced the same problems ourselves, our team and I have been working on Datmo, a tool to help companies more easily and cost-effectively deploy + manage their models in production.

Yeah, the requirements are pretty different than most Data Science teams, especially the very low latency requirements.

The constraints force us to use simple models like linear regression and logistic regression some of the time or at least as a version 1. The inference here is straightforward, multiply and add then take the sigmoid if doing logistic regression.

What we tried to do initially was integrate with C/C++ APIs where possible. We ran into some issues with speed and bugs doing this though, which is why we wrote the inference ourselves. Where we had issues was calling the XGBoost C API from Go. It was extra overhead and too slow. In our benchmarks our implementation in pure Go was many times faster than calling the C API. We also found the multithreaded version to be slower than the single threaded. We found this to be true when calling XGBoost from Java and from Go. We also found this to be true in our own inference implementation it was always faster to walk the trees in a single go routine rather than create some number of worker go routines to walk the trees in parallel.

We were very careful implementing the inference ourselves to make sure the predictions matched. What we did to verify this was create a few toy datasets of about 100 rows with sklearn's make_classification function. We then trained a model using the reference implementation, saved the predictions and the model. We then loaded this model into our implementation and made predictions on the same dataset. We wrote unit tests to compare the predictions and make sure they are the same within some delta. We were able to get our implementation to be within 1e-7 of the reference implementation, in this specific case XGBoost. It was actually more time consuming to deal with parsing the inconsistent JSON model output of XGBoost than it was to implement the GBDT inference algorithm. We also had to make a slight change to the XGBoost code to write out floats to 18 decimal places when writing out the JSON model in order to get the two implementations to match.