Live data from Hacker News

What I Wish I Had Known Before Scaling Uber [video]

youtube.com

231–240 of 284 posts

Re: What I Wish I Had Known Before Scaling Uber [video]

#231
post #215

Earlier quoted context omitted.

Hospital mortality rates in each department are also lowest when there's a conference for that specialty nearby. The doctors all go there, few/no routine surgeries are done that day, far fewer people die.

Isn't that because a surgery, if it fails, will result in the patient dying immediately, rather than some random time later? If you do more surgeries, you will increase both the failed and the successful surgeries, whether your success rate is 10 or 90 percent. Compare: "bomb explosions tend to increase around the time when we send in the bomb squad. I guess bomb defusing is pointless." IOW: Surgeries and bomb defusi…

Surgery in general usually makes patient worse, insurance company's know this, they would rather cash-out a claimant early then pay for a claimant to recover post-surgery.

Expert surgeons, in their eyes the cure for every common cold is 'surgery', like we used to say in UNIX, "If your only tool is a hammer, everything looks like a nail"

A few years ago, in Bend, Oregon at the only Hospital they had the high rate of un-needed back surgery's in the country, upon close inspection it was because all the high-paid back surgeons chose to live there, just about everybody who went to 'st-charles' for anything left dead or in debt from 'back surgery'. The insurance company's finally put an end to it, but it took over ten years, as the AMA also owns the legislature in Salem-OR ( capital of state ).

In summary NEVER go to a doctor, you will die, and if they talk surgery, run like hell.

Re: What I Wish I Had Known Before Scaling Uber [video]

#232

It amazes me they have 1,700 services. It would be hilarious satire to actually see the description of each. And the debugging scenarios he listed make it sound like they have very poor engineers working for them. Who on earth lets an application get to prod that iterates through a list of items and makes a request for each thing? When did we loose our heads and think such an architecture is sane? The UNIX philosophy…

While I'd concede that 1,700 services is a lot, and probably a function of the fact that they're hiring too many engineers too quickly (there's no possible way to indoctrinate people into sane and standardized engineering practices at this sort of hiring growth), I don't actually think what he's describing is that unusual for a company at Uber's scale or experiencing Uber's rate of growth.

I work at Airbnb and while we have many fewer engineers and services than Uber does, most of the issues he talked about in the talk resonated with me to one degree or another.

Re: What I Wish I Had Known Before Scaling Uber [video]

#233
post #161

Earlier quoted context omitted.

Chaos monkey is a simulation, I believe.

It's not a simulation, it's real-world production technology. https://en.wikipedia.org/wiki/Technical_details_of_Netflix

And it's OSS: https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey

Re: What I Wish I Had Known Before Scaling Uber [video]

#234

Earlier quoted context omitted.

Are you implying that software purpose is to provide developers with puzzles to tinker with... in detriment of their paying customers?

I see how one could interpret it that way. I meant to imply rather that software generally needs to be regularly changed in order to continue providing value to its paying customers. At minimum, you've got keeping up with security patches, library/framework deprecations etc. Software which has literally not been changed in years is often an insecure timebomb, with unpatched vulnerabilities, and as soon as you do need…

It's a bad analogy because they don't upgrade ships while they are in use. People aren't saying software is most stable when it's not used, they are saying it's most stable when people stop changing it.

Re: What I Wish I Had Known Before Scaling Uber [video]

#235
post #206

Earlier quoted context omitted.

The point is that unless you use JVM hot reloading (not recommended in Production) you will need to take down your whole app to upgrade that JAR. Now what if your microservice was something minor like a black word filter. Is adding a swear word worthy of a potential outage ?

Is hot reloading that big of a problem?

not for everyone

Re: What I Wish I Had Known Before Scaling Uber [video]

#236

Highly recommended video. Lots of stuff he spoke are very relatable. Like having many repos , storing configs as a separate repo, politics by people, having a tracking system

Sorry, don't think so. I went through the slides and didn't find anything really interesting.

I'm recommending the video explicitly. Not the slides!

Re: What I Wish I Had Known Before Scaling Uber [video]

#237

"Uber is most reliable over the weekends when engineers don't change it" :)

Does anyone who has worked in software for any period of time not realize this? Computers are dumb they (generally) repeat the same tasks reliably _compared to humans_. Occasionally there are system failures but most failure are caused by code or configuration changes, especially when they interact with inputs and systems in untested ways.

Re: What I Wish I Had Known Before Scaling Uber [video]

#238

Earlier quoted context omitted.

I have no idea why people keep thinking microservices is all about scalability. Almost like they've never worked on a problem with them before. Microservices is all about taking a big problem and breaking it down into smaller components, defining the contracts between the components (which an API is), testing the components in isolation and most importantly deploying and running the components independently. It's the…

You don't need to build a distributed system for that. Just build the ShippingServiceLibrary, let others import the jar file (or whatever) and maintain a stable interface. I wrote about this a couple of years ago in more detail, so I'll just reference that: https://www.chrisstucchio.com/blog/2014/microservices_for_th... HN discussion: https://news.ycombinator.com/item?id=8227721

You certainly can do that. A little discussed benefit (or curse) of a microservice is maintaining an API (as opposed to ABI). Changes are slower and people put more thought into the interface as a discrete concern. I am curious if an ideal team would work better with a lib--I think so, but I'm not sure!

Re: What I Wish I Had Known Before Scaling Uber [video]

#239

Earlier quoted context omitted.

And one to aggregate its logs.

And another to monitor whether the logs are being aggregated. And yet another one to ensure that the log-aggregation-monitoring-service is working.

And another to redirect the misspelled first attempt "left-padd-as-service" API calls to the new one

Re: What I Wish I Had Known Before Scaling Uber [video]

#240
post #150

Earlier quoted context omitted.

"Move fast and break things" https://xkcd.com/1428/

Yep. "Move fast and break things" is immature and inappropriate for anything that purports to be reliable. It's fine for toys like social apps, but for serious actual grown-up applications it's a juvenile hindrance.

In an Agile context at least, it means test out ideas and reiterate quickly. Breaking does not necessarily mean breaking existing things in a visible way.
Post reply on HN