Live data from Hacker News

What I Wish I Had Known Before Scaling Uber [video]

youtube.com

11–20 of 284 posts

Re: What I Wish I Had Known Before Scaling Uber [video]

#11
post #4

I didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.

Cab service on a planetary scale :) They have 2000 engineers, 800 microservices and 8000 GIT repositories. Does it still seem trivial to you? EDIT: to downvoters: why shoot the messenger? :)

I'm curious how a company ends up with 10 git repositories per microservice, and 4 repositories per engineer?

Re: What I Wish I Had Known Before Scaling Uber [video]

#12
post #6

"Uber is most reliable over the weekends when engineers don't change it" :)

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

Do they have the same load/users during the weekend?

Re: What I Wish I Had Known Before Scaling Uber [video]

#13
post #4

I didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.

Cab service on a planetary scale :) They have 2000 engineers, 800 microservices and 8000 GIT repositories. Does it still seem trivial to you? EDIT: to downvoters: why shoot the messenger? :)

Sounds like they should sit down and re-think why they need one service for every two engineers (!), and ten repos per service (!!).

Re: What I Wish I Had Known Before Scaling Uber [video]

#14
post #6

"Uber is most reliable over the weekends when engineers don't change it" :)

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

Absolute numbers of alerts are probably a lot less useful than alerts/use, or alerts/users. After all, if people use the services PagerDuty covers less often on the weekend then you might see a lower alert volume even if issues are relatively more common.

Re: What I Wish I Had Known Before Scaling Uber [video]

#15
post #4

I didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.

Cab service on a planetary scale :) They have 2000 engineers, 800 microservices and 8000 GIT repositories. Does it still seem trivial to you? EDIT: to downvoters: why shoot the messenger? :)

This leads to an interesting perspective, that there is a hard limit on how much traffic your services will ever have to handle, and that limit is probably a constant expression on the planet's population.

But I can't imagine how you could possibly need 8000 git repositories unless you're massively over engineering your problem. Project structure tends to reflect the organizations that build them.

Re: What I Wish I Had Known Before Scaling Uber [video]

#16
post #8
post #6

Earlier quoted context omitted.

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

I've come to question releasing often as a result.

I can see the argument that if releasing causes things to break then don't release so frequently, but in practice the end result of that is lots of things breaking at once and having to unpick everything. Debugging is much easier if you're debugging a single change fresh in your mind.

Re: What I Wish I Had Known Before Scaling Uber [video]

#17
post #5
post #4

I didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.

Some cities are bigger than others. That's a heavily imbalanced sharding key.

Why is that a problem? You could count several smaller cities as a single large city, as far as sharding is concerned.

Re: What I Wish I Had Known Before Scaling Uber [video]

#18
post #8
post #6

Earlier quoted context omitted.

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

I've come to question releasing often as a result.

It seems like so many "best practices" are really thinly veiled attempts at exploding complexity with only tenuous potential business advantages.

We create ourselves so many of the problems we are paid to solve.

Re: What I Wish I Had Known Before Scaling Uber [video]

#19
post #17
post #5

Earlier quoted context omitted.

Some cities are bigger than others. That's a heavily imbalanced sharding key.

Why is that a problem? You could count several smaller cities as a single large city, as far as sharding is concerned.

That's like a post office way of thinking about the internet, we already past that point. Plus you haven't solved the problem of managing 800 microservices and 2000 programmers, but now you have a deployment problem also.

Re: What I Wish I Had Known Before Scaling Uber [video]

#20
post #5
post #4

I didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.

Some cities are bigger than others. That's a heavily imbalanced sharding key.

Which isn't necessarily a problem. Just put X shard on the same machine. If the largest city can be handled by a single server, I generally agree with your parent.

I've thought about this problem before, both for a related problem space and with friends working in this specific space. The short version is to create a grid where each square holds car data (id, status, type, x, y, ...) in-memory. Any write-lock of concern is only needed when a car changes grid, and then only on the two grids in questions. This can be layered multiple levels, and your final car-holding structure could be an r-tree or something.

The grid for a city can be sharded across multiple servers. And, if you told me that was necessary, fine..but as-is, I'm suspicious that a pretty basic server can't handle a tens of thousands of cars sending updates every second.

Friends tell me the heavy processing is in routing / map stuff, but this is relatively stateless and can be sent off to a pool of workers to handle.

Post reply on HN