What I Wish I Had Known Before Scaling Uber [video] • Hacker News Board

Re: What I Wish I Had Known Before Scaling Uber [video]

#121

Post by lnanek2 » Wed, Sep 28, 2016, 5:50 PM UTC

mtrpcic wrote: post #110

mranney wrote: post #76

Earlier quoted context omitted.

Our number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances.

Could you give some information as to what the breakdown of functionality is for those services? I can't fathom 1700 different and unique pieces of functionality that would need to be their own services.

I work at a microservice company. An example microservice is our geoservice which simply takes a lat/lon and tells you what service region the user is in (e.g. New York, San Francisco, etc..). You can see how dozens of these services might be needed when handling a single request coming in from the front end or mobile apps. The service may eventually gain another related function or two as we work on tearing down our monolith (internally referred to as the tumor because we do anything to keep it from growing...).

Re: What I Wish I Had Known Before Scaling Uber [video]

#122

Post by iamnothere » Wed, Sep 28, 2016, 5:51 PM UTC

mtberatwork wrote: post #75

iamnothere wrote: post #60

Earlier quoted context omitted.

Is your first example really "naive" though? In my experience, loose, flexible schemas and dynamic languages are very well suited to rapid early-stage development, much more so than rigid languages and schemas. Sure, in the long term things should be refactored, structured, and optimized. But if you do that too soon you risk locking yourself out of potential value, as well as gold-plating things that aren't critical.

> Sure, in the long term things should be refactored, structured, and optimized. How often does that really happen though? Once you've amassed enough technical/data debt, resistance to refactoring increases until it never happens at all. Having well defined, coherent data models and schemas from the start will pay off in the long run. Applications begin and end with data, so why half-ass this from the get go?

I think the key is to do regular refactoring as you go -- it has to be an ingrained part of the process. It's really a management issue. Not every company/project has the foresight to budget for this, of course. If a team can't or won't regularly improve their infrastructure, then yes, a more structured approach would probably be better for anything that needs to last.

Of course there are other considerations. A more "planned" structure always makes sense if you're talking about systems or components that are life-critical or that deal with large flows of money. The "fast and loose" approach makes the most sense when you can tolerate occasional failures, but you have to have fast iterations to be quick-to-market with new features.

Re: What I Wish I Had Known Before Scaling Uber [video]

#123

Post by dsacco » Wed, Sep 28, 2016, 5:53 PM UTC

jackweirdy wrote: post #104

mshenfield wrote: post #55

Earlier quoted context omitted.

Facebook noticed the same thing. http://www.darkcoding.net/software/facebooks-code-quality-pr... - "Exhibit C: Our site works when the engineers go on holiday"

Hospital mortality rates in each department are also lowest when there's a conference for that specialty nearby. The doctors all go there, few/no routine surgeries are done that day, far fewer people die.

This is interesting. Do you have a source for this?

Re: What I Wish I Had Known Before Scaling Uber [video]

#124

Post by kasparsklavins » Wed, Sep 28, 2016, 5:54 PM UTC

dnautics wrote: post #108

Yhippa wrote: post #44

Earlier quoted context omitted.

After working at various enterprises over the years (where deployments are slower in some cases) I've noticed you'd do a Thursday/Friday/weekend deployment, everything "looks good" and you'll still have a bunch of issues Monday morning due to users finally using the system en masse.

Pretty sure the user volume for Uber is higher over the weekends than during the week. Particularly system stressful times are 1:30-2:15am PT on Saturday and Sunday mornings.

There is less commuting on weeekends.

Re: What I Wish I Had Known Before Scaling Uber [video]

#125

Post by 0xmohit » Wed, Sep 28, 2016, 5:58 PM UTC

esrauch wrote: post #11

tiborsaas wrote: post #7

Earlier quoted context omitted.

Cab service on a planetary scale :) They have 2000 engineers, 800 microservices and 8000 GIT repositories. Does it still seem trivial to you? EDIT: to downvoters: why shoot the messenger? :)

I'm curious how a company ends up with 10 git repositories per microservice, and 4 repositories per engineer?

  10 git repositories per microservice

I'm reminded of http://danluu.com/monorepo/

Re: What I Wish I Had Known Before Scaling Uber [video]

#126

Post by 0xmohit » Wed, Sep 28, 2016, 6:00 PM UTC

aristus wrote: post #13

tiborsaas wrote: post #7

Earlier quoted context omitted.

Cab service on a planetary scale :) They have 2000 engineers, 800 microservices and 8000 GIT repositories. Does it still seem trivial to you? EDIT: to downvoters: why shoot the messenger? :)

Sounds like they should sit down and re-think why they need one service for every two engineers (!), and ten repos per service (!!).

Maybe they prefer repos to branches?

Re: What I Wish I Had Known Before Scaling Uber [video]

#127

Post by chejazi » Wed, Sep 28, 2016, 6:04 PM UTC

dnautics wrote: post #108

Yhippa wrote: post #44

Earlier quoted context omitted.

After working at various enterprises over the years (where deployments are slower in some cases) I've noticed you'd do a Thursday/Friday/weekend deployment, everything "looks good" and you'll still have a bunch of issues Monday morning due to users finally using the system en masse.

Pretty sure the user volume for Uber is higher over the weekends than during the week. Particularly system stressful times are 1:30-2:15am PT on Saturday and Sunday mornings.

While a lot of people go out on Friday and Saturday night, I don't think the total volume surpasses commute volume. The surge in pricing you experience late at night is largely due to the number of drivers on the road, which would be less given the time of day.

Edited out consideration: traffic in your area might be a good proxy for Uber volume. Traffic late at night is generally low.

Re: What I Wish I Had Known Before Scaling Uber [video]

#128

Post by agentultra » Wed, Sep 28, 2016, 6:04 PM UTC

What I Wish Small Startups Had Known Before Implementing A Microservices Architecture:

Know your data. Are you serving ~1000 requests per second peak and have room to grow? You're not going to gain much efficiency by introducing engineering complexity, latency, and failure modes.

Best case scenario and your business performs better than expected... does that mean you have a theoretical upper bound in 100k rps? Still not going to gain much.

There are so many well-known strategies for coping with scale that I think the main take-away here for non-Uber companies is to start up-front with some performance characteristics to design for. Set the upper bound on your response times to X ms, over-fill data in order to keep the bound on API queries to 1-2 requests, etc.

Know your data and the program will reveal itself is the rule of thumb I use.

Re: What I Wish I Had Known Before Scaling Uber [video]

#129

Post by enraged_camel » Wed, Sep 28, 2016, 6:04 PM UTC

gjolund wrote: post #39

tiborsaas wrote: post #2

"Uber is most reliable over the weekends when engineers don't change it" :)

"Move fast and break things" is the worst thing to happen to software development that I can remember.

I think it's OK to move fast and break things when you're in the initial product development stage, where you need to develop features quickly and try lots of new stuff to see what sticks. Once you have paying customers though, it may be time to slow down and start caring about stability, at least for your main branch.

Re: What I Wish I Had Known Before Scaling Uber [video]

#130

Post by opendomain » Wed, Sep 28, 2016, 6:05 PM UTC

gtrubetskoy wrote: post #57

I think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to…

No No No. This is like asking which is a better tool - a nail or screw? Since everyone is so familiar with SQL, they are avoiding very good reasons to use NoSQL.

'NoSQL' is just a broad term for datastores that do not normally use standard Structured Query Language to retrieve data. Most NoSQL do allow for for very structured data as well as some query languages that are similar to SQL.

BigTable (Hadoop, Cassandra, Dynamo) and block stores (Amazon S3, Redis, MemcacheD) are absolutely critical to cloud services. Json tuple document DBs are needed for mobile and messaging apps. Graph is for relationships, and Marklogic has an awesome NoSQL Db focused on XML.

Full disclosure: I am the founder of NoSQL.Org - but I also use multiple relational SQL databases every day.