What I Wish I Had Known Before Scaling Uber [video] • Hacker News Board

Re: What I Wish I Had Known Before Scaling Uber [video]

#31

Post by ex_amazon_sde » Wed, Sep 28, 2016, 2:13 PM UTC

aristus wrote: post #13

tiborsaas wrote: post #7

Earlier quoted context omitted.

Cab service on a planetary scale :) They have 2000 engineers, 800 microservices and 8000 GIT repositories. Does it still seem trivial to you? EDIT: to downvoters: why shoot the messenger? :)

Sounds like they should sit down and re-think why they need one service for every two engineers (!), and ten repos per service (!!).

A leading example of overfragmentation.

On-call shifts are going to be interesting with an average of 2.5 engineers per service, not to mention handling people switching teams or leaving the company.

Re: What I Wish I Had Known Before Scaling Uber [video]

#32

Post by andy_ppp » Wed, Sep 28, 2016, 2:14 PM UTC

latch wrote: post #20

xapata wrote: post #5

Earlier quoted context omitted.

Some cities are bigger than others. That's a heavily imbalanced sharding key.

Which isn't necessarily a problem. Just put X shard on the same machine. If the largest city can be handled by a single server, I generally agree with your parent. I've thought about this problem before, both for a related problem space and with friends working in this specific space. The short version is to create a grid where each square holds car data (id, status, type, x, y, ...) in-memory. Any write-lock of conc…

I always wonder in these cases about giving each car an actor in Erlang/Elixir and having complete network transparency, message handling and crashes handled for free.

The routing is very complex too but as you note scales well, until you want to start routing/pickups based on the realtime location of other cars.

Re: What I Wish I Had Known Before Scaling Uber [video]

#33

Post by mason55 » Wed, Sep 28, 2016, 2:17 PM UTC

amelius wrote: post #4

I didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.

I think that only parts can be sharded by city. Anything having to do with users has to be global (because I can visit different cities). I suppose you could try to "move" users from shard to shard but you'd still want a globally unique ID for each user (so you could run cross-city analytics). It feels like you'd be building so much orchestration around making the global parts work in sharded manner that you might as well make the whole thing work globally.

Re: What I Wish I Had Known Before Scaling Uber [video]

#34

Post by JoelBennett » Wed, Sep 28, 2016, 2:20 PM UTC

tiborsaas wrote: post #7

amelius wrote: post #4

I didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.

Cab service on a planetary scale :) They have 2000 engineers, 800 microservices and 8000 GIT repositories. Does it still seem trivial to you? EDIT: to downvoters: why shoot the messenger? :)

This seems almost the exact opposite of what I've heard of Facebook (i.e.: they have a massive repository).

Re: What I Wish I Had Known Before Scaling Uber [video]

#35

Post by amelius » Wed, Sep 28, 2016, 2:25 PM UTC

mason55 wrote: post #33

amelius wrote: post #4

I didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.

I think that only parts can be sharded by city. Anything having to do with users has to be global (because I can visit different cities). I suppose you could try to "move" users from shard to shard but you'd still want a globally unique ID for each user (so you could run cross-city analytics). It feels like you'd be building so much orchestration around making the global parts work in sharded manner that you might as…

> you'd still want a globally unique ID for each user (so you could run cross-city analytics).

Analytics can run in batch mode. Also it is read-only operation. So it is far simpler than designing a distributed database.

As an aside, why would Uber want to reinvent a distributed database for storing user information?

Re: What I Wish I Had Known Before Scaling Uber [video]

#36

Post by dagw » Wed, Sep 28, 2016, 2:26 PM UTC

esrauch wrote: post #11

tiborsaas wrote: post #7

Earlier quoted context omitted.

Cab service on a planetary scale :) They have 2000 engineers, 800 microservices and 8000 GIT repositories. Does it still seem trivial to you? EDIT: to downvoters: why shoot the messenger? :)

I'm curious how a company ends up with 10 git repositories per microservice, and 4 repositories per engineer?

4 repositories per engineer

Is that strange? I've got 6 repos at work for various utilities and "personal" projects I'm working on in addition to the 4 repos for team wide projects.

Re: What I Wish I Had Known Before Scaling Uber [video]

#37

Post by paukiatwee » Wed, Sep 28, 2016, 2:27 PM UTC

Just to confirm, 1000 microservices in this case is 1000 different apps (e.g.different docker images) running simultaneously? 1000 microservices in this case not 1000 microservice instances (e.g. docker instances)?

If it is 1000 microservices as in different apps, then they must have at least 2000 running apps (at least 2 instances per app for HA).

Maybe uber only have 200 "active" microservice app running at the same time where each microservices have N running instances.

I just cant imagine running 1000 different microservices (e.g different docker images, not docker instances) at the same time.

Re: What I Wish I Had Known Before Scaling Uber [video]

#38

Post by teej » Wed, Sep 28, 2016, 2:27 PM UTC

esrauch wrote: post #11

tiborsaas wrote: post #7

Earlier quoted context omitted.

Cab service on a planetary scale :) They have 2000 engineers, 800 microservices and 8000 GIT repositories. Does it still seem trivial to you? EDIT: to downvoters: why shoot the messenger? :)

I'm curious how a company ends up with 10 git repositories per microservice, and 4 repositories per engineer?

I would bet that they have location data in git with each major market being it's own repo.

Re: What I Wish I Had Known Before Scaling Uber [video]

#39

Post by gjolund » Wed, Sep 28, 2016, 2:34 PM UTC

tiborsaas wrote: post #2

"Uber is most reliable over the weekends when engineers don't change it" :)

"Move fast and break things" is the worst thing to happen to software development that I can remember.

Re: What I Wish I Had Known Before Scaling Uber [video]

#40

Post by jfoster » Wed, Sep 28, 2016, 2:37 PM UTC

mason55 wrote: post #33

amelius wrote: post #4

I didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.

I think that only parts can be sharded by city. Anything having to do with users has to be global (because I can visit different cities). I suppose you could try to "move" users from shard to shard but you'd still want a globally unique ID for each user (so you could run cross-city analytics). It feels like you'd be building so much orchestration around making the global parts work in sharded manner that you might as…

To me, it seems the best thing to do in that scenario is taking the huge "shard by city" gain when it comes to rides. Then dealing with the global nature of users by having global user DBs sharded by user ID.