What I Wish I Had Known Before Scaling Uber [video] • Hacker News Board

Re: What I Wish I Had Known Before Scaling Uber [video]

#41

Post by xapata » Wed, Sep 28, 2016, 2:40 PM UTC

latch wrote: post #20

xapata wrote: post #5

Earlier quoted context omitted.

Some cities are bigger than others. That's a heavily imbalanced sharding key.

Which isn't necessarily a problem. Just put X shard on the same machine. If the largest city can be handled by a single server, I generally agree with your parent. I've thought about this problem before, both for a related problem space and with friends working in this specific space. The short version is to create a grid where each square holds car data (id, status, type, x, y, ...) in-memory. Any write-lock of conc…

When the Facebook IPO crashed the NASDAQ, I suspect it's because NASDAQ was sharding by ticker symbol. That was rational until one ticker became 1/3 of the trading volume.

Re: What I Wish I Had Known Before Scaling Uber [video]

#42

Post by eeZah7Ux » Wed, Sep 28, 2016, 2:41 PM UTC

jon-wood wrote: post #25

kowdermeister wrote: post #19

Earlier quoted context omitted.

That's like a post office way of thinking about the internet, we already past that point. Plus you haven't solved the problem of managing 800 microservices and 2000 programmers, but now you have a deployment problem also.

With 800 services and 2000 programmers you already have a deployment problem.

With 800 services and 2000 programmers you already have a problem.

Re: What I Wish I Had Known Before Scaling Uber [video]

#43

Post by adekok » Wed, Sep 28, 2016, 2:42 PM UTC

tiborsaas wrote: post #2

"Uber is most reliable over the weekends when engineers don't change it" :)

I suspect that the issue isn't the engineers. It's the changes.

A system running just fine will usually continue to run just fine. Until it's changed.

Re: What I Wish I Had Known Before Scaling Uber [video]

#44

Post by Yhippa » Wed, Sep 28, 2016, 2:42 PM UTC

kenrose wrote: post #6

tiborsaas wrote: post #2

"Uber is most reliable over the weekends when engineers don't change it" :)

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

After working at various enterprises over the years (where deployments are slower in some cases) I've noticed you'd do a Thursday/Friday/weekend deployment, everything "looks good" and you'll still have a bunch of issues Monday morning due to users finally using the system en masse.

Re: What I Wish I Had Known Before Scaling Uber [video]

#45

Post by anezvigin » Wed, Sep 28, 2016, 2:45 PM UTC

kenrose wrote: post #6

tiborsaas wrote: post #2

"Uber is most reliable over the weekends when engineers don't change it" :)

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

Would PagerDuty consider publishing some anonymized aggregate statistics?

Re: What I Wish I Had Known Before Scaling Uber [video]

#46

Post by _pmf_ » Wed, Sep 28, 2016, 2:49 PM UTC

aristus wrote: post #13

tiborsaas wrote: post #7

Earlier quoted context omitted.

Cab service on a planetary scale :) They have 2000 engineers, 800 microservices and 8000 GIT repositories. Does it still seem trivial to you? EDIT: to downvoters: why shoot the messenger? :)

Sounds like they should sit down and re-think why they need one service for every two engineers (!), and ten repos per service (!!).

> Sounds like they should sit down and re-think why they need one service for every two engineers (!)

They did that; the reason is job security.

Re: What I Wish I Had Known Before Scaling Uber [video]

#47

Post by employee8000 » Wed, Sep 28, 2016, 2:50 PM UTC

kenrose wrote: post #6

tiborsaas wrote: post #2

"Uber is most reliable over the weekends when engineers don't change it" :)

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

90% of outages are caused by configuration changes which is why change management was so hot for enterprise software.

Re: What I Wish I Had Known Before Scaling Uber [video]

#48

Post by lgas » Wed, Sep 28, 2016, 2:50 PM UTC

andy_ppp wrote: post #32

latch wrote: post #20

Earlier quoted context omitted.

Which isn't necessarily a problem. Just put X shard on the same machine. If the largest city can be handled by a single server, I generally agree with your parent. I've thought about this problem before, both for a related problem space and with friends working in this specific space. The short version is to create a grid where each square holds car data (id, status, type, x, y, ...) in-memory. Any write-lock of conc…

I always wonder in these cases about giving each car an actor in Erlang/Elixir and having complete network transparency, message handling and crashes handled for free. The routing is very complex too but as you note scales well, until you want to start routing/pickups based on the realtime location of other cars.

You're saying you want to extend "just let it crash" to the individual car level? :)

Re: What I Wish I Had Known Before Scaling Uber [video]

#49

Post by cortesoft » Wed, Sep 28, 2016, 2:56 PM UTC

asimuvPR wrote: post #8

kenrose wrote: post #6

Earlier quoted context omitted.

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

I've come to question releasing often as a result.

The breakage rate per new feature is fairly constant; if you release 7 new features once a week or 1 new feature once a day, you will have the same number of issues. The question then is; is it easier to deal with all the issues at once, or a smaller number of issues every day?

Re: What I Wish I Had Known Before Scaling Uber [video]

#50

Post by employee8000 » Wed, Sep 28, 2016, 3:05 PM UTC

paukiatwee wrote: post #37

Just to confirm, 1000 microservices in this case is 1000 different apps (e.g.different docker images) running simultaneously? 1000 microservices in this case not 1000 microservice instances (e.g. docker instances)? If it is 1000 microservices as in different apps, then they must have at least 2000 running apps (at least 2 instances per app for HA). Maybe uber only have 200 "active" microservice app running at the sam…

Yes, and much more than 2 per instance of a service.