Live data from Hacker News

What I Wish I Had Known Before Scaling Uber [video]

youtube.com

211–220 of 284 posts

Re: What I Wish I Had Known Before Scaling Uber [video]

#211

"Uber is most reliable over the weekends when engineers don't change it" :)

"A ship in harbor is safe — but that is not what ships are built for."

Are you implying that software purpose is to provide developers with puzzles to tinker with... in detriment of their paying customers?

Re: What I Wish I Had Known Before Scaling Uber [video]

#212

Earlier quoted context omitted.

Could you give some information as to what the breakdown of functionality is for those services? I can't fathom 1700 different and unique pieces of functionality that would need to be their own services.

I work at a microservice company. An example microservice is our geoservice which simply takes a lat/lon and tells you what service region the user is in (e.g. New York, San Francisco, etc..). You can see how dozens of these services might be needed when handling a single request coming in from the front end or mobile apps. The service may eventually gain another related function or two as we work on tearing down our…

I understand what microservices are, but I can't understand what 1700 pieces of unique functionality of Uber could be abstracted into their own services. I am struggling to even think of 100, so I was curious what exactly some of these things were, and how they structured things to need so many service dependencies.

Re: What I Wish I Had Known Before Scaling Uber [video]

#213
post #53

Earlier quoted context omitted.

I've given that kind of model a lot of thought too, but I did not find very satisfying solutions to these problems: - How do you distribute the car actors on nodes, assuming the number of nodes is variable? (I think riak_core looks interesting, but it does not seem to have a way to guarantee unicity of something - it's rather built to replicate the data on multiple nodes for redundancy) - What happens if a node fails…

For the netspit stuff I'd look at Phoenix.Presence works and see how that handles it with a CRDT. There are various ways of distributing things across nodes in Erlang/Elixir but maybe you'd need to build something. I think you are right about the riak core stuff - you could probably keep track of cars using some sort of distributed hash and kill multiple cars if they were to ever spawn. In fact a way of instantiating…

I think that's a cool idea. It would have the downside that error recovery could take a while though, depending on the permdown period, so during that time a driver would be stuck; while in a request-based system they can immediately try again and it would work (hit a different instance). But it might not be too bad.

Re: What I Wish I Had Known Before Scaling Uber [video]

#214
post #8
post #6

Earlier quoted context omitted.

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

I've come to question releasing often as a result.

Funny, I've taken it as justification for releasing often. If I have a hard time changing one thing without breaking the service, it's nearly impossible to change a hundred things without breaking something. Since it's a given that I will have to change things, I'll try to stick to a scope where I stand a chance of doing so successfully.

Re: What I Wish I Had Known Before Scaling Uber [video]

#215

Earlier quoted context omitted.

Facebook noticed the same thing. http://www.darkcoding.net/software/facebooks-code-quality-pr... - "Exhibit C: Our site works when the engineers go on holiday"

Hospital mortality rates in each department are also lowest when there's a conference for that specialty nearby. The doctors all go there, few/no routine surgeries are done that day, far fewer people die.

Isn't that because a surgery, if it fails, will result in the patient dying immediately, rather than some random time later? If you do more surgeries, you will increase both the failed and the successful surgeries, whether your success rate is 10 or 90 percent.

Compare: "bomb explosions tend to increase around the time when we send in the bomb squad. I guess bomb defusing is pointless."

IOW: Surgeries and bomb defusing tend to move forward a lot of the death-probability-mass (while, in theory, destroying some of it).

Re: What I Wish I Had Known Before Scaling Uber [video]

#216
post #134

Earlier quoted context omitted.

Taken in a healthy organization, Conway's Law is not a bad thing. It is really just more of an observation. So... taken to this example, it sounds like the company is a confusing mess of people trying to figure out who they need to coordinate with to make something happen.

> It is really just more of an observation. Or maybe some instruction into how to assemble the organization.

True. I was using it as an argument to adjust the system architecture, as well. My argument was it didn't matter really which changed to get things into alignment, but that having them be different was a bit of a concern.

Re: What I Wish I Had Known Before Scaling Uber [video]

#217

Earlier quoted context omitted.

I'm not sure if it's ignorance that leads people to dismiss things they don't understand so much as it is a coping mechanism similar to that which gave birth to various religions. Ignorance just lets it thrive after the birth.

Phrasing aside, it's a legitimate question. I've worked at a really really large web behemoth serving orders of magnitude more users and many many unique and disparate products (as opposed to a handful products that Uber serves). If I counted all of their production related services , I'm not sure they'd amount to anywhere close to 1700. Now I know "microservices" is the new hotness, but surely there are limits to hu…

I have heard of, but never witnessed, groups where the team stays together but cycles through projects a sprint at a time. One or two teams keeping three or four projects spinning but making the projects take turns.

I don't know if they provide popcorn at the meetings where the project managers explain why they deserve the next sprint.

Re: What I Wish I Had Known Before Scaling Uber [video]

#218

Earlier quoted context omitted.

I'm not sure if it's ignorance that leads people to dismiss things they don't understand so much as it is a coping mechanism similar to that which gave birth to various religions. Ignorance just lets it thrive after the birth.

Phrasing aside, it's a legitimate question. I've worked at a really really large web behemoth serving orders of magnitude more users and many many unique and disparate products (as opposed to a handful products that Uber serves). If I counted all of their production related services , I'm not sure they'd amount to anywhere close to 1700. Now I know "microservices" is the new hotness, but surely there are limits to hu…

Nonsense. libc? A microservice for every function call! Surely if you can wrap your head around a library you can wrap it around a thousand little microservices.

(Snark exists in this comment.)

Re: What I Wish I Had Known Before Scaling Uber [video]

#219
post #164

Earlier quoted context omitted.

No, that had absolutely nothing to do with it. It was based on a bad assumption about volume of trade entries/cancellations during the initial price calculation.

Sounds like you know more. Care to explain what the assumption was and how it caused the malfunction?

http://www.system-logic.com/commentary/2013/06/04/NASDAQ-and...

Re: What I Wish I Had Known Before Scaling Uber [video]

#220

Earlier quoted context omitted.

> I have no idea why people keep thinking microservices is all about scalability. It's an aspect. It's often the beginning of a micro-service migration story in talks I've heard. > Microservices is all about taking a big problem and breaking it down into smaller components ... and putting the network between them. It's all well and good but the tradeoffs are not obvious there either. Most engineers I know who claim t…

You seem to be repeating these weird myths that have no basis in reality. You don't need to be an expert in distributed systems to use microservices. It's literally replacing a function call with an RPC call. That's it. If you want to make tracing easier you tag the user request with an ID and pass it through your API calls or use something like Zipkin. But needing formal verification in order to test your architectu…

> You seem to be repeating these weird myths that have no basis in reality.

No basis at all? I knew I was unhinged...

> You don't need to be an expert in distributed systems to use microservices.

True. Hooray for abstractions. You don't need to understand how the V8 engine allocates and garbage collects memory either... well until you do.

> It's literally replacing a function call with an RPC call.

You're not wrong.

Which is the point. Whether for architectural or performance reasons I think you need to understand your domain and model your data first. For domains that map really well to the microservice architecture you're not going to have many problems.

And a formal specification is overkill for many, many scenarios. That doesn't mean they're useless. They're just not useful, perhaps, for e-commerce sites.

But anywhere you have an RPC call that depends on external state, ordering, consensus... the point is that the tradeoffs are not always immediately apparent unless you know your data really well.

> And I've worked on 2 of the world's top 5 ecommerce sites which both use microservices.

And I've worked on public and private clouds! Cool.

The point was and still is the same whether performance or architecture... think about your data! The rest falls out from that.

Post reply on HN