What I Wish I Had Known Before Scaling Uber [video] • Hacker News Board

Re: What I Wish I Had Known Before Scaling Uber [video]

#131

Post by taeric » Wed, Sep 28, 2016, 6:06 PM UTC

lnanek2 wrote: post #121

mtrpcic wrote: post #110

Earlier quoted context omitted.

Could you give some information as to what the breakdown of functionality is for those services? I can't fathom 1700 different and unique pieces of functionality that would need to be their own services.

I work at a microservice company. An example microservice is our geoservice which simply takes a lat/lon and tells you what service region the user is in (e.g. New York, San Francisco, etc..). You can see how dozens of these services might be needed when handling a single request coming in from the front end or mobile apps. The service may eventually gain another related function or two as we work on tearing down our…

This just sounds crazy to me. How often do the services that are responsible for a given region change?

Re: What I Wish I Had Known Before Scaling Uber [video]

#132

Post by taeric » Wed, Sep 28, 2016, 6:07 PM UTC

Roritharr wrote: post #67

I'm not a big green-IT guy, but always pushing your systems close to its load maxima and then backing off with the test traffic as real traffic comes in feels like an enormous waste of electricity.

Almost as much as the overhead of splitting everything to a thousand microservices...

Re: What I Wish I Had Known Before Scaling Uber [video]

#133

Post by weberc2 » Wed, Sep 28, 2016, 6:08 PM UTC

mshenfield wrote: post #55

tiborsaas wrote: post #2

"Uber is most reliable over the weekends when engineers don't change it" :)

Facebook noticed the same thing. http://www.darkcoding.net/software/facebooks-code-quality-pr... - "Exhibit C: Our site works when the engineers go on holiday"

"Move fast and break things" https://xkcd.com/1428/

Re: What I Wish I Had Known Before Scaling Uber [video]

#134

Post by taeric » Wed, Sep 28, 2016, 6:09 PM UTC

sah2ed wrote: post #69

esrauch wrote: post #11

Earlier quoted context omitted.

I'm curious how a company ends up with 10 git repositories per microservice, and 4 repositories per engineer?

That's just a manifestation of Conway's Law [1]. [1] https://en.wikipedia.org/wiki/Conway's_law

Taken in a healthy organization, Conway's Law is not a bad thing. It is really just more of an observation. So... taken to this example, it sounds like the company is a confusing mess of people trying to figure out who they need to coordinate with to make something happen.

Re: What I Wish I Had Known Before Scaling Uber [video]

#135

Post by admiralhack_ » Wed, Sep 28, 2016, 6:12 PM UTC

Great video. I went in expecting it to cover mostly the technical side of things. Instead Matt gave a great overview of the team / engineering organization dynamics to watch out for when adopting microservices. (I particularly liked that he pointed out how developers may write new code / services instead of dealing with team politics.)

Re: What I Wish I Had Known Before Scaling Uber [video]

#136

Post by jackweirdy » Wed, Sep 28, 2016, 6:13 PM UTC

dsacco wrote: post #123

jackweirdy wrote: post #104

Earlier quoted context omitted.

Hospital mortality rates in each department are also lowest when there's a conference for that specialty nearby. The doctors all go there, few/no routine surgeries are done that day, far fewer people die.

This is interesting. Do you have a source for this?

I believe I read it here, though there are several similar studies and this might not be the one I originally read.

https://www.ncbi.nlm.nih.gov/pubmed/25531231

Re: What I Wish I Had Known Before Scaling Uber [video]

#137

Post by ChuckMcM » Wed, Sep 28, 2016, 6:16 PM UTC

kenrose wrote: post #6

tiborsaas wrote: post #2

"Uber is most reliable over the weekends when engineers don't change it" :)

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

That conclusion is well founded. We correlated issues at Blekko across a lot of different factors, the one that always held was code or configuration changes. Not too surprising in the large but definitely confirmed by the data.

Re: What I Wish I Had Known Before Scaling Uber [video]

#138

Post by oldmanjay » Wed, Sep 28, 2016, 6:19 PM UTC

cagataygurturk wrote: post #96

paukiatwee wrote: post #37

Just to confirm, 1000 microservices in this case is 1000 different apps (e.g.different docker images) running simultaneously? 1000 microservices in this case not 1000 microservice instances (e.g. docker instances)? If it is 1000 microservices as in different apps, then they must have at least 2000 running apps (at least 2 instances per app for HA). Maybe uber only have 200 "active" microservice app running at the sam…

Maybe my ignorance, but do they have services like left-pad-as-a microservice? I can't understand which 1000 microservices you can derive for a cab-renting application.

I'm not sure if it's ignorance that leads people to dismiss things they don't understand so much as it is a coping mechanism similar to that which gave birth to various religions. Ignorance just lets it thrive after the birth.

Re: What I Wish I Had Known Before Scaling Uber [video]

#139

Post by Scarblac » Wed, Sep 28, 2016, 6:21 PM UTC

lnanek2 wrote: post #121

mtrpcic wrote: post #110

Earlier quoted context omitted.

Could you give some information as to what the breakdown of functionality is for those services? I can't fathom 1700 different and unique pieces of functionality that would need to be their own services.

I work at a microservice company. An example microservice is our geoservice which simply takes a lat/lon and tells you what service region the user is in (e.g. New York, San Francisco, etc..). You can see how dozens of these services might be needed when handling a single request coming in from the front end or mobile apps. The service may eventually gain another related function or two as we work on tearing down our…

Why would you make that a separate service when e.g. one query on a PostGIS table can do that?

Re: What I Wish I Had Known Before Scaling Uber [video]

#140

Post by Scarblac » Wed, Sep 28, 2016, 6:22 PM UTC

dastbe wrote: post #90

jrochkind1 wrote: post #85

Earlier quoted context omitted.

that sounds nightmarish

Why? I'm assuming their highish engineer/service ratio is because their services do less individually. Anecdotally, I've worked on services that ran tens of thousands of instances across the world. You build the tools to manage them and it works very well.

> Why?

People talk about interpreted languages being slow sometimes, now your program is divided over 1700 separate servers and instead of keeping variables in memory you have to serialize and send them over the network all the time.