Live data from Hacker News

What I Wish I Had Known Before Scaling Uber [video]

youtube.com

141–150 of 284 posts

Re: What I Wish I Had Known Before Scaling Uber [video]

#141

Earlier quoted context omitted.

Maybe my ignorance, but do they have services like left-pad-as-a microservice? I can't understand which 1000 microservices you can derive for a cab-renting application.

left-pad-as-a microservice You can also have another service to monitor the left-pad service.

And one to aggregate its logs.

Re: What I Wish I Had Known Before Scaling Uber [video]

#142

Earlier quoted context omitted.

Facebook noticed the same thing. http://www.darkcoding.net/software/facebooks-code-quality-pr... - "Exhibit C: Our site works when the engineers go on holiday"

Hospital mortality rates in each department are also lowest when there's a conference for that specialty nearby. The doctors all go there, few/no routine surgeries are done that day, far fewer people die.

That study gave no causal reasons. In particular, there was no effect on number of surgeries done. It's just as possible that research-oriented doctors go to the conference, and those doctors are worse at performing surgeries.

Re: What I Wish I Had Known Before Scaling Uber [video]

#143
post #39

Earlier quoted context omitted.

"Move fast and break things" is the worst thing to happen to software development that I can remember.

I think it's OK to move fast and break things when you're in the initial product development stage, where you need to develop features quickly and try lots of new stuff to see what sticks. Once you have paying customers though, it may be time to slow down and start caring about stability, at least for your main branch.

It's also ok to "move fast and break things" when the most critical thing you're breaking is your annoying high school "friend"s ability to share the latest racist meme with you. There's a reason facebook made this phrase popular.

Re: What I Wish I Had Known Before Scaling Uber [video]

#144

I think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to…

(3) people who ponder what's the best solution for a given situation? Sometimes you want dynamic languages, or schema-less data. Sometimes you want rigid interfaces and verifiability.

Eg rolling out a public API in Thrift/protobuf will severely difficult its adoption, whereas Rest/JSON is pretty much the standard - but then building microservices that communicate to each other in Rest/JSON quickly leads to a costly, hard to maintain, inconsistent mess.

We're obsessed with "one fits all" absolutes in tech. We should have more "it depends" imho

Re: What I Wish I Had Known Before Scaling Uber [video]

#145
post #11

Earlier quoted context omitted.

I'm curious how a company ends up with 10 git repositories per microservice, and 4 repositories per engineer?

I have over 3 dozen private Git repos for personal amusement or utility projects for stuff that professional consumer software doesn't work as well as what I want. Then I have almost at least 10 repos on GutHub (not this userid). At my last work, 9 month contract, I had 4 I think, aside from our main team repos just to manage personal info documents, my bin folder, etc. It's easy for one developer to have several. On…

THE HN PC police are back, up to their old antics.

Re: What I Wish I Had Known Before Scaling Uber [video]

#146
post #18
post #8

Earlier quoted context omitted.

I've come to question releasing often as a result.

It seems like so many "best practices" are really thinly veiled attempts at exploding complexity with only tenuous potential business advantages. We create ourselves so many of the problems we are paid to solve.

"We create ourselves so many of the problems we are paid to solve"

Sounds lucrative ;)

Re: What I Wish I Had Known Before Scaling Uber [video]

#147
post #43

"Uber is most reliable over the weekends when engineers don't change it" :)

I suspect that the issue isn't the engineers. It's the changes. A system running just fine will usually continue to run just fine. Until it's changed.

I hear what you're saying, in theory. In practice, a system is always changing – regardless of whether developers are deploying code changes.

Some events (or "changes") I've seen in last year that caused a system to stop running fine:

- analyst ran query that locked reporting DB, which had ripple effects

- integration partner "tweaked" schema of data uploaded to us

- NIC started dropping packets

- growing data made backup software exceed expected window, locking files

- critical security patch released (didn't cause a problem exactly, but did change the system from having no known security holes to having a big one)

On and on. So, yeah, I'm not disputing the idea that changes are often the source of problems. I'm just saying that any moderately complex system is constantly encountering new events whether or not the developers are making changes.

This isn't a rebuttal of your statement so much, as it is rebuttal of a common attitude I see in business folks. A lot of non-technical executives seem to have the mentality that software is "done" after it's built, which is naive IMO.

Active systems require active maintenance. You can avoid a lot of problems with intelligent architecture and robust instrumentation/monitoring, but at the end of the day systems rot and will eventually stop running fine.

And even if your perfectly planned and architected system runs in total isolation on a private server where security isn't a major concern, you'll still build up technical debt if you aren't routinely upgrading major libraries, etc. You'll get to a point where you want to implement some feature, and while there are 3 excellent OSS libraries for your platform to accomplish it, none of them is compatible with your 5 year old version.

You probably know all of this and might even agree with most of it. I just had a visceral reaction to the idea that "a system running just fine will usually continue to run just fine," and felt compelled to respond. :)

Note: most of my experience is with public or enterprise web applications. I imagine other types of systems have other problems.

Re: What I Wish I Had Known Before Scaling Uber [video]

#148

Earlier quoted context omitted.

left-pad-as-a microservice You can also have another service to monitor the left-pad service.

And one to aggregate its logs.

And another to monitor whether the logs are being aggregated.

And yet another one to ensure that the log-aggregation-monitoring-service is working.

Re: What I Wish I Had Known Before Scaling Uber [video]

#149

Earlier quoted context omitted.

Maybe my ignorance, but do they have services like left-pad-as-a microservice? I can't understand which 1000 microservices you can derive for a cab-renting application.

I'm not sure if it's ignorance that leads people to dismiss things they don't understand so much as it is a coping mechanism similar to that which gave birth to various religions. Ignorance just lets it thrive after the birth.

Phrasing aside, it's a legitimate question. I've worked at a really really large web behemoth serving orders of magnitude more users and many many unique and disparate products (as opposed to a handful products that Uber serves). If I counted all of their production related services, I'm not sure they'd amount to anywhere close to 1700. Now I know "microservices" is the new hotness, but surely there are limits to human cognition that limit the total number of moving pieces any team can manage without maintaining a constant state of crisis.

It'll be interesting to know the number of people oncall at any given time and the number of prod alerts per hour/day/week.

Re: What I Wish I Had Known Before Scaling Uber [video]

#150

Earlier quoted context omitted.

Facebook noticed the same thing. http://www.darkcoding.net/software/facebooks-code-quality-pr... - "Exhibit C: Our site works when the engineers go on holiday"

"Move fast and break things" https://xkcd.com/1428/

Yep. "Move fast and break things" is immature and inappropriate for anything that purports to be reliable. It's fine for toys like social apps, but for serious actual grown-up applications it's a juvenile hindrance.
Post reply on HN