Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Machine Learning at Uber with Michelangelo

eng.uber.com

Post reply on HN
Print view View original

41–50 of 61 posts

Re: Scaling Machine Learning at Uber with Michelangelo

#41

Post by jquery »

marmaduke wrote: post #37
jquery wrote: post #33

Earlier quoted context omitted.

Yes, Uber suffers from original sin, but you don't become the fastest growing company of all time by avoiding any toe-stepping. Look at the pathetic state of "Jump Bikes" in SF. Uber is judiciously following all the regulations these days (in all areas), and SF is "generously" upping the Jump Bike limit from roughly 200 to 400 total bikes. It's really a pathetic number of bikes and doesn't come close to meeting deman…

> you don't become the fastest growing company of all time by avoiding any toe-stepping this is purified hubris, how can you not vomit on your keyboard while writing that? > roughly 200 to 400 total bikes. It's really a pathetic number of bikes and doesn't come close to meeting demand maybe the mandate of SF is not to satisfy demand or Uber's profit incentive but to keep public interest in mind, e.g. ensure that Uber…

>this is purified hubris, how can you not vomit on your keyboard while writing that?

I'm curious how you get around. Do you own a car? It's a fairly privileged view that only the well-off should have access to point-to-point transportation and maybe some civil disobedience was in order to rectify this injustice.

>maybe the mandate of SF is not to satisfy demand or Uber's profit incentive but to keep public interest in mind, e.g. ensure that Uber doesn't develop a monopoly on whatever a jump bike is.

Prior restraint on free enterprise that lacks negative externalities opens the door to crony capitalism replete with bribes, donations, and rent-seeking. In government, never ascribe to benevolence that which can be better explained by greed or power-seeking.

Re: Scaling Machine Learning at Uber with Michelangelo

#42

Post by sandGorgon »

marmaduke wrote: post #27
sandGorgon wrote: post #17

Earlier quoted context omitted.

Thanks for your reply. Actually the question was more around "how do you create your models and what do you mean treating them as code", "why slurm and not something like airflow" , "what is the test/performance setup - backtesting, smoke test" etc etc The Gitlab stuff is easier to understand.

Ah right, > how do you create your models and what do you mean treating them as code we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data, where we assess performance with LOO approximations for Bayesian models (notably PSIS) and some labeling from experts (which is not t…

super interesting. thanks for sharing.

>we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data

This is the part that everyone seems reinventing. Have you looked at PyML (https://eng.uber.com/michelangelo-pyml/). What are some of your learnings around jupyter -> production code. A lot of these are around conventions - "write a function called train(), fit(), test()". Is that the basis of your pipeline as well ?

Re: Scaling Machine Learning at Uber with Michelangelo

#43

Post by marmaduke »

sandGorgon wrote: post #42
marmaduke wrote: post #27

Earlier quoted context omitted.

Ah right, > how do you create your models and what do you mean treating them as code we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data, where we assess performance with LOO approximations for Bayesian models (notably PSIS) and some labeling from experts (which is not t…

super interesting. thanks for sharing. > we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data This is the part that everyone seems reinventing. Have you looked at PyML ( https://eng.uber.com/michelangelo-pyml/ ). What are some of your learnings around jupyter -> productio…

It’s not so simple for our models (hierarchical Bayesian time series models, often nonlinear, which may not be typical): we spend a lot of time digging through the data itself, forward simulations of model, and refactoring/tweaking model structure. PyML (as described in the link you provided) doesn’t appear to support the first two parts, which are prerequisites to improving the model IMO.

Usually when we are doing more of the train/fit/test cycle, there’s an argparse script to quickly try different parameter values succinctly (which is run and tracked by the above CI setup)

I wouldn’t say we’re reinventing since a better solution isn’t very clear (though PyML et al look interesting)

edit forward simulation isn't a frequent thing in posts on generic ML algorithms, so just as an example: suppose you run a model and see an oscillatory component along a temporal dimensions in your residual error, and you add a oscillatory component to your model, and rerun it but still see a residual with an oscillation. You can run a forward simulation of your model to see what frequency it's predicting and check against what's seen in the data, and fix it. This is a contrived example but when you have multiple competing priors or model components, this is an effective way to debug their behavior.

Re: Scaling Machine Learning at Uber with Michelangelo

#44

Post by marmaduke »

sandGorgon wrote: post #42
marmaduke wrote: post #27

Earlier quoted context omitted.

Ah right, > how do you create your models and what do you mean treating them as code we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data, where we assess performance with LOO approximations for Bayesian models (notably PSIS) and some labeling from experts (which is not t…

super interesting. thanks for sharing. > we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data This is the part that everyone seems reinventing. Have you looked at PyML ( https://eng.uber.com/michelangelo-pyml/ ). What are some of your learnings around jupyter -> productio…

this article (https://towardsdatascience.com/uber-introduces-pyml-their-se...) does a better job motivating PyML, or maybe I'm just more awake now. In any case, I see what you mean. The GitLab CI setup we have builds Docker images out of our models, and we use branch names to target datasets, so "production" usage is "just" creating a branch, watching it run, checking results, etc.

Maybe a missing detail is that our models are run-once, once results are QA'd, they are sent to relevant practitioner, so Uber's query-per-second stuff is irrelevant for us (for now), which I can see simplifies the deployment question enormously.

Re: Scaling Machine Learning at Uber with Michelangelo

#46

Post by dsumenkovic »

marmaduke wrote: post #44
sandGorgon wrote: post #42

Earlier quoted context omitted.

super interesting. thanks for sharing. > we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data This is the part that everyone seems reinventing. Have you looked at PyML ( https://eng.uber.com/michelangelo-pyml/ ). What are some of your learnings around jupyter -> productio…

this article ( https://towardsdatascience.com/uber-introduces-pyml-their-se... ) does a better job motivating PyML, or maybe I'm just more awake now. In any case, I see what you mean. The GitLab CI setup we have builds Docker images out of our models, and we use branch names to target datasets, so "production" usage is "just" creating a branch, watching it run, checking results, etc. Maybe a missing detail is that ou…

Hello, Community Advocate from GitLab here. I was reading through your comments and it's great to hear how you use GitLab for your setup. Thanks for sharing your story with the community and we'd love to hear more from you on how GitLab helps you.

Re: Scaling Machine Learning at Uber with Michelangelo

#48

Post by marmaduke »

dsumenkovic wrote: post #46
marmaduke wrote: post #44

Earlier quoted context omitted.

this article ( https://towardsdatascience.com/uber-introduces-pyml-their-se... ) does a better job motivating PyML, or maybe I'm just more awake now. In any case, I see what you mean. The GitLab CI setup we have builds Docker images out of our models, and we use branch names to target datasets, so "production" usage is "just" creating a branch, watching it run, checking results, etc. Maybe a missing detail is that ou…

Hello, Community Advocate from GitLab here. I was reading through your comments and it's great to hear how you use GitLab for your setup. Thanks for sharing your story with the community and we'd love to hear more from you on how GitLab helps you.

> we'd love to hear more from you on how GitLab helps you.

do you have specific questions?

Re: Scaling Machine Learning at Uber with Michelangelo

#49

Post by marmaduke »

jquery wrote: post #41
marmaduke wrote: post #37

Earlier quoted context omitted.

> you don't become the fastest growing company of all time by avoiding any toe-stepping this is purified hubris, how can you not vomit on your keyboard while writing that? > roughly 200 to 400 total bikes. It's really a pathetic number of bikes and doesn't come close to meeting demand maybe the mandate of SF is not to satisfy demand or Uber's profit incentive but to keep public interest in mind, e.g. ensure that Uber…

>this is purified hubris, how can you not vomit on your keyboard while writing that? I'm curious how you get around. Do you own a car? It's a fairly privileged view that only the well-off should have access to point-to-point transportation and maybe some civil disobedience was in order to rectify this injustice. >maybe the mandate of SF is not to satisfy demand or Uber's profit incentive but to keep public interest i…

> ...

all off topic generalities.

Re: Scaling Machine Learning at Uber with Michelangelo

#50

Post by sandGorgon »

marmaduke wrote: post #44
sandGorgon wrote: post #42

Earlier quoted context omitted.

super interesting. thanks for sharing. > we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data This is the part that everyone seems reinventing. Have you looked at PyML ( https://eng.uber.com/michelangelo-pyml/ ). What are some of your learnings around jupyter -> productio…

this article ( https://towardsdatascience.com/uber-introduces-pyml-their-se... ) does a better job motivating PyML, or maybe I'm just more awake now. In any case, I see what you mean. The GitLab CI setup we have builds Docker images out of our models, and we use branch names to target datasets, so "production" usage is "just" creating a branch, watching it run, checking results, etc. Maybe a missing detail is that ou…

Interesting. In that case, why do you even use Docker ? Does it simplify distribution of models easier ?

Would love to know more about your packaging setup - the branch name to divide datasets is a nice trick (I'll use it as well).

How does your CI know where to find models ? Im betting you are using some kind of convention here - one model per py file...so package each py file in a docker container.

If it is possible, would love to see the skeleton structure of one of your pre-packaged files.

Tldr - it seems you invented something like pyml as well. Are the deployment scripts+model skeletons open source ?

Post reply on HN