Ask HN: What's Your CI/CD Workflow for Your Machine Learning Projects? • Hacker News Board

Re: Ask HN: What's Your CI/CD Workflow for Your Machine Learning Projects?

#21

Post by claytonjy » Tue, Mar 06, 2018, 8:00 PM UTC

volker48 wrote: post #19

claytonjy wrote: post #13

Earlier quoted context omitted.

Thanks for this; I really appreciate the detail here. There seems to be a lack of these kinds of explanations around. One piece did give me a bit of surprise: You might also need to implement the inference code as well to get the speed you need I've never had the super-low-latency requirements you have, but as you point out this seems amazingly error-prone. I'd love to hear anything else you can share about the cost-…

Yeah, the requirements are pretty different than most Data Science teams, especially the very low latency requirements. The constraints force us to use simple models like linear regression and logistic regression some of the time or at least as a version 1. The inference here is straightforward, multiply and add then take the sigmoid if doing logistic regression. What we tried to do initially was integrate with C/C++…

This is all SO fascinating to me. Multiple threads slowing stuff down & 18 decimal places being relevant stick out as surprising.

Part of me is thankful to not have these problems, while another part thinks it'd be a lot of fun to do this kind of last-mile engineering.

Re: Ask HN: What's Your CI/CD Workflow for Your Machine Learning Projects?

#22

Post by chewxy » Tue, Mar 06, 2018, 8:06 PM UTC

claytonjy wrote: post #13

volker48 wrote: post #6

Having put many models into production in an almost real time environment (ad servers that need predictions in First, I would highly recommend wrapping your ML models in some kind of microservice. Depending on your production requirements and if the ML is in Python a fairly simple Flask/Sanic web server should be sufficient. This is great because you can leave all your feature transformation code as is in Python. If…

Thanks for this; I really appreciate the detail here. There seems to be a lack of these kinds of explanations around. One piece did give me a bit of surprise: You might also need to implement the inference code as well to get the speed you need I've never had the super-low-latency requirements you have, but as you point out this seems amazingly error-prone. I'd love to hear anything else you can share about the cost-…

I have similar experience with GP, which is how I ended up writing Gorgonia (https://github.com/gorgonia/gorgonia). Serialization of models is easy: .npy files are very good formats though I know a number of people who will disagree - I find they typically prefer pb as a serialization format, which I think is better for over-the-wire not for storage

Also like GP, I used to work in advertising. The low latency bits are because RTB servers want you to respond within a certain amount of time. When I was in advertising, our solution was to precalculate a whole bunch of things, and throw them into redis. It was then a simple lookup of the hash of a vector (for bidding related stuff)

Re: Ask HN: What's Your CI/CD Workflow for Your Machine Learning Projects?

#23

Post by chewxy » Tue, Mar 06, 2018, 8:08 PM UTC

volker48 wrote: post #19

claytonjy wrote: post #13

Earlier quoted context omitted.

Thanks for this; I really appreciate the detail here. There seems to be a lack of these kinds of explanations around. One piece did give me a bit of surprise: You might also need to implement the inference code as well to get the speed you need I've never had the super-low-latency requirements you have, but as you point out this seems amazingly error-prone. I'd love to hear anything else you can share about the cost-…

Yeah, the requirements are pretty different than most Data Science teams, especially the very low latency requirements. The constraints force us to use simple models like linear regression and logistic regression some of the time or at least as a version 1. The inference here is straightforward, multiply and add then take the sigmoid if doing logistic regression. What we tried to do initially was integrate with C/C++…

You wrote XGBoost in Go? I've been looking for that and dreading writing one

Re: Ask HN: What's Your CI/CD Workflow for Your Machine Learning Projects?

#24

Post by asampat » Tue, Mar 06, 2018, 8:27 PM UTC

We faced the issue of building CV models a lot as grad students but at that time reliability wasn't really something we had to solve for. Once we had to implement them for industrial applications we found we had to ensure there was reproducibility and versioning throughout the process. Now that we have put a few computer vision algorithms into production we decided to architect our code in the following way.

1) training code is written with normal packages (we tend to prefer keras with a tensorflow backend), these are trained on our own GPUs since this is often the cheapest and are done in bulk --> side note is that TPUs/GPUs may one day be better but certainly too expensive currently.

2) prediction / inference code will also be written in the similar packages but will be tied with the final weights files that we get from the model

3) deployment code -- in order to enable a reliable system we have used celery for distributed queueing and converting our prediction functions into celery tasks which can then me passed to workers that can process and return the result to an API endpoint. This allows us to scale our workloads as needed with throughput (depending on request requirements)

This architecture allows us to test during training time using our training code and validation sets, while also enabling testing of different models versions through our prediction APIs. We often would just write scripts for testing that we then run via our CI/CD workflow.

Tip: keep your code as simple as possible and don't rewrite your code unless your throughput requirements mandate it. good error handling will go a long ways here and likely will be easier if written in a language you're most familiar with.

Re: Ask HN: What's Your CI/CD Workflow for Your Machine Learning Projects?

#25

Post by volker48 » Tue, Mar 06, 2018, 8:49 PM UTC

chewxy wrote: post #23

volker48 wrote: post #19

Earlier quoted context omitted.

Yeah, the requirements are pretty different than most Data Science teams, especially the very low latency requirements. The constraints force us to use simple models like linear regression and logistic regression some of the time or at least as a version 1. The inference here is straightforward, multiply and add then take the sigmoid if doing logistic regression. What we tried to do initially was integrate with C/C++…

You wrote XGBoost in Go? I've been looking for that and dreading writing one

I wrote the inference step of XGBoost in Go. It will make predictions after loading in an XGBoost JSON model. Writing the training portion of XGBoost would be much harder.

Re: Ask HN: What's Your CI/CD Workflow for Your Machine Learning Projects?

#26

Post by volker48 » Tue, Mar 06, 2018, 8:53 PM UTC

claytonjy wrote: post #21

volker48 wrote: post #19

Earlier quoted context omitted.

Yeah, the requirements are pretty different than most Data Science teams, especially the very low latency requirements. The constraints force us to use simple models like linear regression and logistic regression some of the time or at least as a version 1. The inference here is straightforward, multiply and add then take the sigmoid if doing logistic regression. What we tried to do initially was integrate with C/C++…

This is all SO fascinating to me. Multiple threads slowing stuff down & 18 decimal places being relevant stick out as surprising. Part of me is thankful to not have these problems, while another part thinks it'd be a lot of fun to do this kind of last-mile engineering.

Yeah, I had a very hard time believing that the multithreaded approach would be slower. Its so counterintuitive since at first blush it seems that walking N trees is an embarrassingly parallel problem. I tested up to 1000 trees and single threaded was still faster. I'm sure at some point the multithreaded approach will win out, but its beyond the number of trees and max depth we are using.

Re: Ask HN: What's Your CI/CD Workflow for Your Machine Learning Projects?

#27

Post by claytonjy » Tue, Mar 06, 2018, 9:06 PM UTC

volker48 wrote: post #26

claytonjy wrote: post #21

Earlier quoted context omitted.

This is all SO fascinating to me. Multiple threads slowing stuff down & 18 decimal places being relevant stick out as surprising. Part of me is thankful to not have these problems, while another part thinks it'd be a lot of fun to do this kind of last-mile engineering.

Yeah, I had a very hard time believing that the multithreaded approach would be slower. Its so counterintuitive since at first blush it seems that walking N trees is an embarrassingly parallel problem. I tested up to 1000 trees and single threaded was still faster. I'm sure at some point the multithreaded approach will win out, but its beyond the number of trees and max depth we are using.

I've half-convinced myself it's because we're talking about GBM's and not Random Forests (where my mind goes first). One of the smart things about XGBoost is parallelizing training by multithreading the variable selection at each node, but that doesn't apply to inference; I imagine you gotta predict trees sequentially since each takes the previous output as an input? Now I wonder what those extra threads were even doing...

Re: Ask HN: What's Your CI/CD Workflow for Your Machine Learning Projects?

#28

Post by lee101 » Tue, Mar 06, 2018, 10:43 PM UTC

Hi i'm founder of https://bitbank.nz a crypto currency live prediction dashboard/API/bulk data service.

Our system streams in market data from exchanges, creates forecasts with python/sk-learn and displays the data, we also have background processes that updates our accuracy over time once real data is available.

We test our code with the normal python unit/integration/end to end testing methods locally with local copies of all of our components (except firebase for live UI updates, we don't have a dev version of that yet just use live forecast data when testing the UI charts/display), would probably get expensive/cumbersome to setup dev environments with local firebases in them.

with deployment we simply ssh into machines, git pull latest code and supervisorctl restart so its fairly low tech, the forecaster has a roughly minute outage when we deploy new models because there is a decent process that computes and caches a data structure of historical features for use in the forecaster.

In terms of maintaining a reliable online stream of data input, feature computation/prediction pipeline we run the code under supervisor aswell as running a manager process under supervisor, that manager process checks if conditions are turning bad (OOM/no progress updates by the forecaster) and restarts things if anything goes wrong.

For testing we also use the standard training/test data split when running backtesting/machine learning optimisation algorithms to train parameters of the algorithm. If things perform better on training and test data over a long enough time period to build confidence then we will deploy a new model.

Using graphite/graphana to monitor prediction accuracy over time is a good idea as mentioned already :) and some kind of alerting/monitoring if things go down.

Re: Ask HN: What's Your CI/CD Workflow for Your Machine Learning Projects?

#29

Post by agibsonccc » Wed, Mar 07, 2018, 6:22 AM UTC

Disclaimer: I compete in this space and may compete with your internal team, your cloud vendor, or something you are interested in.

FWIW: Production is an overloaded term. CI may not even be applicable here. Say you're doing batch inference where you need to run jobs every 24 hours on a large amount of data: That might be tied to some cron job.

That being said you could use a CI system for that in theory.

There are also other factors here: What other kind of things do you want to track? Experiments results? Wrong results by your machine learning algorithm?

Concisely: What kind of deployment requirements do you have and what are your goals?

If you are edoing real time, what does "deployment" even mean? Are you serving in real time via a rest api? Are you doing streaming? What are your throughput requirements? What about latency? Is that even hooked up to a CI system?

Something that is vaguely related: How do you test the accuracy of different models across your cluster? Say you want to do a self deployment, what if you want to tie that to say: a workspace where you produced the results?

Is that hooked up to a CI system? If so, what's your use case?

Then there's that common hand off from data scientist to production, what does that look like? A sibling thread mentioned some of these things.

If anyone else is curious about this stuff, we deploy deep learning models in locked down environments both on kubernetes as well as touching hadoop clusters. Happy to answer questions.

Re: Ask HN: What's Your CI/CD Workflow for Your Machine Learning Projects?

#30

Post by shabazp » Wed, Mar 07, 2018, 7:14 AM UTC

shabazp wrote: post #17

I have deployed ML algorithms into production including computer vision and data science models. From my experience, it's based on the application and cost we are comfortable with. 1. Keras/Tensorflow based algorithm(Applicable for any compute intensive or GPU-capable algorithm): Deployed the method (as a flask service) inside a Docker container along with a queueing system(for reliability w/ redis). We can now decid…

3. Another important component for CI/CD is the integration of tools like Airflow in order to schedule and monitor workflows. This helps us in deploying newly trained models.