What I Wish I Had Known Before Scaling Uber [video] • Hacker News Board

Re: What I Wish I Had Known Before Scaling Uber [video]

#111

Post by pc86 » Wed, Sep 28, 2016, 5:35 PM UTC

danek wrote: post #68

cortesoft wrote: post #49

Earlier quoted context omitted.

The breakage rate per new feature is fairly constant; if you release 7 new features once a week or 1 new feature once a day, you will have the same number of issues. The question then is; is it easier to deal with all the issues at once, or a smaller number of issues every day?

In addition to that, I would also worry about interactions between issues. I tend to lean towards spreading out the issues over time to make the eventual diagnosis easier. Occasionally we'll have a problem where we cannot deploy to production for several days (normally it's once a day). A massive inventory of ready-to-deploy features builds up. When we do finally deploy, this deluge of features and fixes creates new…

Sounds like a testing/UAT issue.

Re: What I Wish I Had Known Before Scaling Uber [video]

#112

Post by dastbe » Wed, Sep 28, 2016, 5:39 PM UTC

softawre wrote: post #102

dastbe wrote: post #90

Earlier quoted context omitted.

Why? I'm assuming their highish engineer/service ratio is because their services do less individually. Anecdotally, I've worked on services that ran tens of thousands of instances across the world. You build the tools to manage them and it works very well.

It's not the number of instances that he's talking about, it's the number of images. Microservices can go to far. I'm very thankful for this video.

The # of services vs # of engineers isn't really bad though; you just need the tools to make the commonalities between services easy.

Re: What I Wish I Had Known Before Scaling Uber [video]

#113

Post by petetnt » Wed, Sep 28, 2016, 5:41 PM UTC

Really enjoyed this talk. Our services don't quite (yet :)) run at that scale, but many of the issues mentioned have already peaked at some point. It's also good to have (more) validation to some choices we have made in the past, are currently making or are thinking about making in short term future.

Re: What I Wish I Had Known Before Scaling Uber [video]

#114

Post by hueving » Wed, Sep 28, 2016, 5:41 PM UTC

mason55 wrote: post #33

amelius wrote: post #4

I didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.

I think that only parts can be sharded by city. Anything having to do with users has to be global (because I can visit different cities). I suppose you could try to "move" users from shard to shard but you'd still want a globally unique ID for each user (so you could run cross-city analytics). It feels like you'd be building so much orchestration around making the global parts work in sharded manner that you might as…

Everything related to driver selection and real time tracking is very specific to one location. The server responsible for tracking drivers in San Jose doesn't need to reference rider locations in NYC under any circumstance.

The things you highlighted are very light load comparatively. Sign in, load account info, then dispatch to location specific shard.

Re: What I Wish I Had Known Before Scaling Uber [video]

#115

Post by jweir » Wed, Sep 28, 2016, 5:43 PM UTC

selckin wrote: post #12

kenrose wrote: post #6

Earlier quoted context omitted.

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

Do they have the same load/users during the weekend?

No, in the video he states that the weekend is their busiest period.

Re: What I Wish I Had Known Before Scaling Uber [video]

#116

Post by seangrogg » Wed, Sep 28, 2016, 5:44 PM UTC

mtberatwork wrote: post #75

iamnothere wrote: post #60

Earlier quoted context omitted.

Is your first example really "naive" though? In my experience, loose, flexible schemas and dynamic languages are very well suited to rapid early-stage development, much more so than rigid languages and schemas. Sure, in the long term things should be refactored, structured, and optimized. But if you do that too soon you risk locking yourself out of potential value, as well as gold-plating things that aren't critical.

> Sure, in the long term things should be refactored, structured, and optimized. How often does that really happen though? Once you've amassed enough technical/data debt, resistance to refactoring increases until it never happens at all. Having well defined, coherent data models and schemas from the start will pay off in the long run. Applications begin and end with data, so why half-ass this from the get go?

Assuming you're not omniscient you'll be refactoring regardless. The difference is whether you'll be paying as you go (clients want a JSON API, we need to add new columns to a table but it'll lock rows) or if you'll be taking on technical debt to be repaid in the future (turns out Mongo sucks and we would do much better with Cassandra for serious horizontal scaling).

I believe that if you aren't extremely certain about what the future holds it may be best to work with a more flexible technology first and transition to a more structured setup once you have solved for your problems and identified intended future features. And if you are extremely certain about what the future holds you're either insanely good at your job or just insane.

Re: What I Wish I Had Known Before Scaling Uber [video]

#117

Post by asimuvPR » Wed, Sep 28, 2016, 5:44 PM UTC

jon-wood wrote: post #16

asimuvPR wrote: post #8

Earlier quoted context omitted.

I've come to question releasing often as a result.

I can see the argument that if releasing causes things to break then don't release so frequently, but in practice the end result of that is lots of things breaking at once and having to unpick everything. Debugging is much easier if you're debugging a single change fresh in your mind.

You are right. Making small changes that can be isolated for debugging purposes is a good approach. What I mean is that we should always question "best" practices and how we apply them to our development process. These days there is a tendency to drink the kool-aid (guilty of this as well). DevOps is something that we are still learning and developing as a profession.

Re: What I Wish I Had Known Before Scaling Uber [video]

#118

Post by hueving » Wed, Sep 28, 2016, 5:44 PM UTC

xapata wrote: post #41

latch wrote: post #20

Earlier quoted context omitted.

Which isn't necessarily a problem. Just put X shard on the same machine. If the largest city can be handled by a single server, I generally agree with your parent. I've thought about this problem before, both for a related problem space and with friends working in this specific space. The short version is to create a grid where each square holds car data (id, status, type, x, y, ...) in-memory. Any write-lock of conc…

When the Facebook IPO crashed the NASDAQ, I suspect it's because NASDAQ was sharding by ticker symbol. That was rational until one ticker became 1/3 of the trading volume.

No, that had absolutely nothing to do with it. It was based on a bad assumption about volume of trade entries/cancellations during the initial price calculation.

Re: What I Wish I Had Known Before Scaling Uber [video]

#119

Post by seangrogg » Wed, Sep 28, 2016, 5:47 PM UTC

eric_the_read wrote: post #64

gtrubetskoy wrote: post #57

I think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to…

Amusingly, I've just finished trying to convince a team using C in their firmware to use Protobufs; they were trying to convince me to use JSON. Oh, and my servers are in Ruby.

Protobuf v3 allows use of JSON and gRPC for Node.js handles JavaScript objects quite handily. Might not be mutually exclusive.

Re: What I Wish I Had Known Before Scaling Uber [video]

#120

Post by 0xmohit » Wed, Sep 28, 2016, 5:49 PM UTC

cagataygurturk wrote: post #96

paukiatwee wrote: post #37

Just to confirm, 1000 microservices in this case is 1000 different apps (e.g.different docker images) running simultaneously? 1000 microservices in this case not 1000 microservice instances (e.g. docker instances)? If it is 1000 microservices as in different apps, then they must have at least 2000 running apps (at least 2 instances per app for HA). Maybe uber only have 200 "active" microservice app running at the sam…

Maybe my ignorance, but do they have services like left-pad-as-a microservice? I can't understand which 1000 microservices you can derive for a cab-renting application.

  left-pad-as-a microservice

You can also have another service to monitor the left-pad service.