Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Machine Learning at Uber with Michelangelo

eng.uber.com

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Scaling Machine Learning at Uber with Michelangelo

#51

Post by marmaduke »

cocobongo wrote: post #38
marmaduke wrote: post #36

Earlier quoted context omitted.

> I think Uber drivers/couriers are freelancers yep so they should be compensated more, not less, in sight of the precarity of their job > When you factor in that many (probably most) Uber drivers are only working part-time, that's significant income from a super flexible job super flexible for whom? the drivers? or for Uber? There are two sides to the gig style work, one side is a corp that's got teams of PhDs and a…

Freelancers are compensated whatever the market values them at. If they can get better pay/work elsewhere, then they do. Driving for Uber is as flexible job as it gets. I don't see how it's flexible for Uber - Uber can only offer a ride if a driver decides out of their own free will to accept a ride. For many people (2+ Million), driving for Uber is worth the money. They decided that it's a better gig than their othe…

Begging for money in the street is even more flexible — no car required — but I doubt you’d say those doing so are satisfied.

> free will to accept a ride

Whenever money is involved, free will goes out the window. It’s naive to think of all these people as rational actors.

Re: Scaling Machine Learning at Uber with Michelangelo

#52

Post by marmaduke »

sandGorgon wrote: post #50
marmaduke wrote: post #44

Earlier quoted context omitted.

this article ( https://towardsdatascience.com/uber-introduces-pyml-their-se... ) does a better job motivating PyML, or maybe I'm just more awake now. In any case, I see what you mean. The GitLab CI setup we have builds Docker images out of our models, and we use branch names to target datasets, so "production" usage is "just" creating a branch, watching it run, checking results, etc. Maybe a missing detail is that ou…

Interesting. In that case, why do you even use Docker ? Does it simplify distribution of models easier ? Would love to know more about your packaging setup - the branch name to divide datasets is a nice trick (I'll use it as well). How does your CI know where to find models ? Im betting you are using some kind of convention here - one model per py file...so package each py file in a docker container. If it is possibl…

Our GitLab instance has a lot of projects and it’s been helpful for the users to have a set of template projects each with their own Docker image. Some of those images are many gigabytes in size, tricky env vars etc. Docker “democratized” CI for most of our scientific personnel who aren’t devs, since they can hit the Fork button and have a working CI config to base their project on.

In the ML projects, it serves mainly to package dependencies, and to ensure some basic security constraints: raw datasets are accessible read only, ensuring that if we suspect some issue with cached results (cause our inner orchestrator is Make..) we can nuke all the results and start over from scratch, sure the raw data is intact.

The models and arguments are in the CI config. No magic there, but since it’s all in the repo I’m ok with it.

This whole setup was put together for an upcoming clinical trial as steps toward ISO quality norms compliance, and I can’t share it now. I do intend to reproduce it in an open form alongside our existing software (GitHub.com/the-virtual-brain) when it’s ready.

In any case I appreciate your questions a lot: they drove me to think a little harder and see why stuff like Michelango and PyML is stuff that even we (academic/clinical) group should be using... if we can find the time to do it.

Re: Scaling Machine Learning at Uber with Michelangelo

#53

Post by mlthoughts2018 »

marmaduke wrote: post #16
mlthoughts2018 wrote: post #14

Earlier quoted context omitted.

Why would you do this instead of using pymc3?

PyMC3 didn’t run well on GPUs last I tried. That may have changed but I find PyTorch easier to work with than Theano or TensorFlow.

Just in case other readers stumble by, neither of these perceptions of pymc is accurate.

GPU operability is well-supported, and much like Keras, pymc provides well-designed abstractions over top of TensorFlow, making the downsides of raw TensorFlow mostly irrelevant.

I like PyTorch a lot too, but any time I see someone say PyTorch is easier than TensorFlow, it usually just means that person only tried PyTorch, learned some special knowledge about it, and now they don’t want to admit using a different framework might be the better choice, even if it requires giving up some of what’s nice about PyTorch.

Re: Scaling Machine Learning at Uber with Michelangelo

#54

Post by dsumenkovic »

marmaduke wrote: post #48
dsumenkovic wrote: post #46

Earlier quoted context omitted.

Hello, Community Advocate from GitLab here. I was reading through your comments and it's great to hear how you use GitLab for your setup. Thanks for sharing your story with the community and we'd love to hear more from you on how GitLab helps you.

> we'd love to hear more from you on how GitLab helps you. do you have specific questions?

We wanted to hear what features you like using the most and how do those features help you with setting up your project. However, you wrote https://news.ycombinator.com/item?id=18384804 which answers a lot of the questions. Thanks!

Re: Scaling Machine Learning at Uber with Michelangelo

#55

Post by marmaduke »

mlthoughts2018 wrote: post #53
marmaduke wrote: post #16

Earlier quoted context omitted.

PyMC3 didn’t run well on GPUs last I tried. That may have changed but I find PyTorch easier to work with than Theano or TensorFlow.

Just in case other readers stumble by, neither of these perceptions of pymc is accurate. GPU operability is well-supported, and much like Keras, pymc provides well-designed abstractions over top of TensorFlow, making the downsides of raw TensorFlow mostly irrelevant. I like PyTorch a lot too, but any time I see someone say PyTorch is easier than TensorFlow, it usually just means that person only tried PyTorch, learne…

That’s a fairly aggressive response.

Both TF and Theano require static graph while PyTorch lets you use Python’s regular control flows (if, for, while, etc). This makes building modular model components much easier, since you can reason about execution mostly as if it’s normal numerical Python code.

I have tried running PyMC3 models on GPUs (when they were on Theano; not sure if they have transitioned since) and it is slower than CPUs, not for small models but the big, SIMD-wide ones. When I ported the same thing to Pyro/PyTorch, it was clearly making good use of the GPU, not bottlenecked by useless CPU-GPU transfers

Maybe that’s changed now, so as they say the only useful benchmark is your own code.

Re: Scaling Machine Learning at Uber with Michelangelo

#56

Post by mlthoughts2018 »

marmaduke wrote: post #55
mlthoughts2018 wrote: post #53

Earlier quoted context omitted.

Just in case other readers stumble by, neither of these perceptions of pymc is accurate. GPU operability is well-supported, and much like Keras, pymc provides well-designed abstractions over top of TensorFlow, making the downsides of raw TensorFlow mostly irrelevant. I like PyTorch a lot too, but any time I see someone say PyTorch is easier than TensorFlow, it usually just means that person only tried PyTorch, learne…

That’s a fairly aggressive response. Both TF and Theano require static graph while PyTorch lets you use Python’s regular control flows (if, for, while, etc). This makes building modular model components much easier, since you can reason about execution mostly as if it’s normal numerical Python code. I have tried running PyMC3 models on GPUs (when they were on Theano; not sure if they have transitioned since) and it i…

> “I have tried running PyMC3 models on GPUs (when they were on Theano; not sure if they have transitioned since) and it is slower than CPUs, not for small models but the big, SIMD-wide ones.“

Can you post a link to your code with some synthetic data of the sizes you’re talking about to demonstrate this? I hear it as a criticism a lot, but have never found it to be true (full disclosure: I work on a large-scale production system that uses pymc for huge Bayesian logistic regression and huge hierarchical models, both in GPU mode out of necessity).

> “Both TF and Theano require static graph while PyTorch lets you use Python’s regular control flows (if, for, while, etc). This makes building modular model components much easier, since you can reason about execution mostly as if it’s normal numerical Python code.”

I can’t tell if you’ve looked into pymc or not based on this (or Keras either for that matter), since in pymc, GPU mode is just a Theano setting, you don’t actually write any Theano code, manipulate any graphs or sessions directly, or anything else. You just call pm.sample with the appropriate mode settings at it is executed on the GPU.

Much like with Keras, where you can also easily use Python native control flow, context managers and so on, pymc doesn’t require low-level usage of underlying computation graph abstractions.

Again, I really like PyTorch too, but people just seem to have only ever tried PyTorch, liked one or two things about it, forgive the parts that are bad about it (like needing to explicitly write a wrapper for the backwards calculation for custom layers, which you don’t need to do in Keras for example), and generalize to criticize other tools.

Re: Scaling Machine Learning at Uber with Michelangelo

#57

Post by p1esk »

mlthoughts2018 wrote: post #56
marmaduke wrote: post #55

Earlier quoted context omitted.

That’s a fairly aggressive response. Both TF and Theano require static graph while PyTorch lets you use Python’s regular control flows (if, for, while, etc). This makes building modular model components much easier, since you can reason about execution mostly as if it’s normal numerical Python code. I have tried running PyMC3 models on GPUs (when they were on Theano; not sure if they have transitioned since) and it i…

> “I have tried running PyMC3 models on GPUs (when they were on Theano; not sure if they have transitioned since) and it is slower than CPUs, not for small models but the big, SIMD-wide ones.“ Can you post a link to your code with some synthetic data of the sizes you’re talking about to demonstrate this? I hear it as a criticism a lot, but have never found it to be true (full disclosure: I work on a large-scale produ…

like needing to explicitly write a wrapper for the backwards calculation for custom layers, which you don’t need to do in Keras for example

Not sure I understand - you will need to write a backwards pass regardless if you use Keras, PyTorch, or anything else. With Keras, you would need to modify the underlying backend code (e.g. with tf.RegisterGradient or tf.custom_gradient). With Pytorch you write the backward() function, which is about the same amount of effort.

Re: Scaling Machine Learning at Uber with Michelangelo

#58

Post by mlthoughts2018 »

p1esk wrote: post #57
mlthoughts2018 wrote: post #56

Earlier quoted context omitted.

> “I have tried running PyMC3 models on GPUs (when they were on Theano; not sure if they have transitioned since) and it is slower than CPUs, not for small models but the big, SIMD-wide ones.“ Can you post a link to your code with some synthetic data of the sizes you’re talking about to demonstrate this? I hear it as a criticism a lot, but have never found it to be true (full disclosure: I work on a large-scale produ…

like needing to explicitly write a wrapper for the backwards calculation for custom layers, which you don’t need to do in Keras for example Not sure I understand - you will need to write a backwards pass regardless if you use Keras, PyTorch, or anything else. With Keras, you would need to modify the underlying backend code (e.g. with tf.RegisterGradient or tf.custom_gradient). With Pytorch you write the backward() fu…

You missed the point entirely. When you compose operations in Keras, it automatically generates the backpropagation implementation, you do not need RegisterGradient, custom_gradient or anything else if you are making new operations or layers as the composition of existing operations (whether that is logical indexing, concatenation, math functions, whatever).

In PyTorch, you still do have to define the backward function and worry about bookkeeping the gradient, clearing gradient values at the appropriate time, and explicitly calling to calculate these things in verbose optimizer invocation code.

I encourage you to check out how this works in Keras, because it is simply just factually different than what you are saying, in ways that are specifically designed to remove certain types of boilerplate or overhead or bookkeeping that are required by PyTorch.

Re: Scaling Machine Learning at Uber with Michelangelo

#59

Post by p1esk »

mlthoughts2018 wrote: post #58
p1esk wrote: post #57

Earlier quoted context omitted.

like needing to explicitly write a wrapper for the backwards calculation for custom layers, which you don’t need to do in Keras for example Not sure I understand - you will need to write a backwards pass regardless if you use Keras, PyTorch, or anything else. With Keras, you would need to modify the underlying backend code (e.g. with tf.RegisterGradient or tf.custom_gradient). With Pytorch you write the backward() fu…

You missed the point entirely. When you compose operations in Keras, it automatically generates the backpropagation implementation, you do not need RegisterGradient, custom_gradient or anything else if you are making new operations or layers as the composition of existing operations (whether that is logical indexing, concatenation, math functions, whatever). In PyTorch, you still do have to define the backward functi…

No, you're wrong about Pytorch. If your custom op is a combination of existing ops, you don't need to define a custom backward pass. This is true for any DL framework with autodiff. For more details, look at this answer [1].

Regarding more verbose Pytorch code for the update step, compare:

In Tensorflow:

loss = tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=output_logits)

optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)

sess.run(optimizer)

In PyTorch:

optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

loss = nn.CrossEntropyLoss()(output, label)

optimizer.zero_grad()

loss.backward()

optimizer.step()

In my opinion, PyTorch makes the parameter update process a lot easier to understand, control, and modify (if needed). For example what if you want to modify gradients right before the weight update? In PyTorch I'd do it right here in my code after the loss.backward() statement, while in TF I'd have to modify the optimizer code. Which option would you prefer?

[1] https://stackoverflow.com/questions/44428784/when-is-a-pytor...

Re: Scaling Machine Learning at Uber with Michelangelo

#60

Post by marmaduke »

mlthoughts2018 wrote: post #58
p1esk wrote: post #57

Earlier quoted context omitted.

like needing to explicitly write a wrapper for the backwards calculation for custom layers, which you don’t need to do in Keras for example Not sure I understand - you will need to write a backwards pass regardless if you use Keras, PyTorch, or anything else. With Keras, you would need to modify the underlying backend code (e.g. with tf.RegisterGradient or tf.custom_gradient). With Pytorch you write the backward() fu…

You missed the point entirely. When you compose operations in Keras, it automatically generates the backpropagation implementation, you do not need RegisterGradient, custom_gradient or anything else if you are making new operations or layers as the composition of existing operations (whether that is logical indexing, concatenation, math functions, whatever). In PyTorch, you still do have to define the backward functi…

> PyTorch, you still do have to define the backward function and worry about bookkeeping the gradient, clearing gradient values at the appropriate time, and explicitly calling to calculate these things in verbose optimizer invocation code

I’ve definitely never had to do that. Where do you get this from?

Post reply on HN