Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What I Wish I Had Known Before Scaling Uber [video]

youtube.com

Post reply on HN
Print view View original

151–160 of 284 posts

Re: What I Wish I Had Known Before Scaling Uber [video]

#151

Post by dnautics »

chejazi wrote: post #127
dnautics wrote: post #108

Earlier quoted context omitted.

Pretty sure the user volume for Uber is higher over the weekends than during the week. Particularly system stressful times are 1:30-2:15am PT on Saturday and Sunday mornings.

While a lot of people go out on Friday and Saturday night, I don't think the total volume surpasses commute volume. The surge in pricing you experience late at night is largely due to the number of drivers on the road, which would be less given the time of day. Edited out consideration: traffic in your area might be a good proxy for Uber volume. Traffic late at night is generally low.

Your statement is true in San Francisco, but Pacific time zone also encompasses Oakland, south bay, los Angeles, and san Diego, where relatively few people commute using Uber compared to going out using Uber.

From personal experience, having driven for Lyft and Uber I'm two cities (Sf, sd) surge in SF does get acutely high in the morning, more broadly in the evening (work departure times are spread out over a wide range of hours), and acute around 2am. In California, bar closures statewide are 2am at the latest, so that accounts for unified departure times in multiple markets.

Re: What I Wish I Had Known Before Scaling Uber [video]

#152

Post by hinkley »

kenrose wrote: post #6
tiborsaas wrote: post #2

"Uber is most reliable over the weekends when engineers don't change it" :)

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

I'm bad at optimism, but maybe the takeaway here is that we're finally at the point where it's not the hardware or systems that are the biggest problem. The automation is actually working well enough that it's not the tall tent pole.

Re: What I Wish I Had Known Before Scaling Uber [video]

#153

Post by hinkley »

ChuckMcM wrote: post #137
kenrose wrote: post #6

Earlier quoted context omitted.

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

That conclusion is well founded. We correlated issues at Blekko across a lot of different factors, the one that always held was code or configuration changes. Not too surprising in the large but definitely confirmed by the data.

So I should fire all my developers is what you're saying...

Man, I'm gonna save so much money.

Re: What I Wish I Had Known Before Scaling Uber [video]

#154

Post by eternalban »

I think these "alarming trends" are highlighting that operational complexity is an easier pill to take than conceptual complexity, for most workers in the field.

Microservices address that gap.

And in the process the field is transformed from one of software developers to software operators. This generation is witnessing the transfer of the IT crew from the "back office" to the "boiler room".

Re: What I Wish I Had Known Before Scaling Uber [video]

#155

Post by _puk »

mrinterweb wrote: post #97
novaleaf wrote: post #82

Earlier quoted context omitted.

Camps 1 and 2 are not mutually exclusive (well, except your inflammatory "naively" comment). Rest/JSON is a well understood, broadly adopted, low friction RPC format. NoSQL is not always MongoDB (for example, Google Datastore is ACID compliant), and schema enforcement via an ORM layer I would argue is actually a good thing, as it provides schema validation at compile time.

Rest/JSON is not a RPC. Technologies like Thrift can be used for RPC. It depends on how you define low friction for rest + JSON. JSON is schemaless, and that can be great for prototyping, but as soon as deployed services get out of sync, in terms of how they communicate, it becomes more of a burden than an advantage. Thrift, protobuf, avro, can enforce schemas and can raise exceptions for communication mismatches so…

JSON is not inherently schemaless, it's just generally not used with one.

Initiatives like JSONSchema go some way to restoring some constraints to the format and prevents unchecked deviation over time.

Re: What I Wish I Had Known Before Scaling Uber [video]

#156

Post by majormajor »

eternalban wrote: post #154

I think these "alarming trends" are highlighting that operational complexity is an easier pill to take than conceptual complexity, for most workers in the field. Microservices address that gap. And in the process the field is transformed from one of software developers to software operators. This generation is witnessing the transfer of the IT crew from the "back office" to the "boiler room".

Do you think that's it's that people would rather deal with operational complexity, or just that it's easier to not think about operational complexity early on ("running locally, with low volume, everything works together great!")?

Personally, I vastly prefer debugging and building on non-microservice architectures than on something split willy-nilly into dozens of services (because most implementations I've seen don't do microservices with clean conceptual boundaries - it's more political/organizational divisions that determine what lands where, not architectural concerns).

Re: What I Wish I Had Known Before Scaling Uber [video]

#157

Post by [deleted] »

dagw wrote: post #36
esrauch wrote: post #11

Earlier quoted context omitted.

I'm curious how a company ends up with 10 git repositories per microservice, and 4 repositories per engineer?

4 repositories per engineer Is that strange? I've got 6 repos at work for various utilities and "personal" projects I'm working on in addition to the 4 repos for team wide projects.

[deleted]

Re: What I Wish I Had Known Before Scaling Uber [video]

#158

Post by AYBABTME »

asimuvPR wrote: post #8
kenrose wrote: post #6

Earlier quoted context omitted.

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

I've come to question releasing often as a result.

Or the opposite: embrace failure, practice resiliency, work with it instead of against it!

Re: What I Wish I Had Known Before Scaling Uber [video]

#159

Post by ryandrake »

hinkley wrote: post #153
ChuckMcM wrote: post #137

Earlier quoted context omitted.

That conclusion is well founded. We correlated issues at Blekko across a lot of different factors, the one that always held was code or configuration changes. Not too surprising in the large but definitely confirmed by the data.

So I should fire all my developers is what you're saying... Man, I'm gonna save so much money.

You laugh, but a lot of time the best thing for a software product is to not make so many damn changes. People tend to over-estimate the value of features and tweaks and under-estimate the risk of things going wrong.
Post reply on HN