"Uber is most reliable over the weekends when engineers don't change it" :)
"A ship in harbor is safe — but that is not what ships are built for."
What I Wish I Had Known Before Scaling Uber [video]
211–220 of 284 posts
Re: What I Wish I Had Known Before Scaling Uber [video]
#212Earlier quoted context omitted.
Could you give some information as to what the breakdown of functionality is for those services? I can't fathom 1700 different and unique pieces of functionality that would need to be their own services.
I work at a microservice company. An example microservice is our geoservice which simply takes a lat/lon and tells you what service region the user is in (e.g. New York, San Francisco, etc..). You can see how dozens of these services might be needed when handling a single request coming in from the front end or mobile apps. The service may eventually gain another related function or two as we work on tearing down our…
Re: What I Wish I Had Known Before Scaling Uber [video]
#213Earlier quoted context omitted.
I've given that kind of model a lot of thought too, but I did not find very satisfying solutions to these problems: - How do you distribute the car actors on nodes, assuming the number of nodes is variable? (I think riak_core looks interesting, but it does not seem to have a way to guarantee unicity of something - it's rather built to replicate the data on multiple nodes for redundancy) - What happens if a node fails…
For the netspit stuff I'd look at Phoenix.Presence works and see how that handles it with a CRDT. There are various ways of distributing things across nodes in Erlang/Elixir but maybe you'd need to build something. I think you are right about the riak core stuff - you could probably keep track of cars using some sort of distributed hash and kill multiple cars if they were to ever spawn. In fact a way of instantiating…
Re: What I Wish I Had Known Before Scaling Uber [video]
#214Earlier quoted context omitted.
We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.
I've come to question releasing often as a result.
Re: What I Wish I Had Known Before Scaling Uber [video]
#215Earlier quoted context omitted.
Facebook noticed the same thing. http://www.darkcoding.net/software/facebooks-code-quality-pr... - "Exhibit C: Our site works when the engineers go on holiday"
Hospital mortality rates in each department are also lowest when there's a conference for that specialty nearby. The doctors all go there, few/no routine surgeries are done that day, far fewer people die.
Compare: "bomb explosions tend to increase around the time when we send in the bomb squad. I guess bomb defusing is pointless."
IOW: Surgeries and bomb defusing tend to move forward a lot of the death-probability-mass (while, in theory, destroying some of it).
Re: What I Wish I Had Known Before Scaling Uber [video]
#216Earlier quoted context omitted.
Taken in a healthy organization, Conway's Law is not a bad thing. It is really just more of an observation. So... taken to this example, it sounds like the company is a confusing mess of people trying to figure out who they need to coordinate with to make something happen.
> It is really just more of an observation. Or maybe some instruction into how to assemble the organization.
Re: What I Wish I Had Known Before Scaling Uber [video]
#217Earlier quoted context omitted.
I'm not sure if it's ignorance that leads people to dismiss things they don't understand so much as it is a coping mechanism similar to that which gave birth to various religions. Ignorance just lets it thrive after the birth.
Phrasing aside, it's a legitimate question. I've worked at a really really large web behemoth serving orders of magnitude more users and many many unique and disparate products (as opposed to a handful products that Uber serves). If I counted all of their production related services , I'm not sure they'd amount to anywhere close to 1700. Now I know "microservices" is the new hotness, but surely there are limits to hu…
I don't know if they provide popcorn at the meetings where the project managers explain why they deserve the next sprint.
Re: What I Wish I Had Known Before Scaling Uber [video]
#218Earlier quoted context omitted.
I'm not sure if it's ignorance that leads people to dismiss things they don't understand so much as it is a coping mechanism similar to that which gave birth to various religions. Ignorance just lets it thrive after the birth.
Phrasing aside, it's a legitimate question. I've worked at a really really large web behemoth serving orders of magnitude more users and many many unique and disparate products (as opposed to a handful products that Uber serves). If I counted all of their production related services , I'm not sure they'd amount to anywhere close to 1700. Now I know "microservices" is the new hotness, but surely there are limits to hu…
(Snark exists in this comment.)
Re: What I Wish I Had Known Before Scaling Uber [video]
#219Earlier quoted context omitted.
No, that had absolutely nothing to do with it. It was based on a bad assumption about volume of trade entries/cancellations during the initial price calculation.
Sounds like you know more. Care to explain what the assumption was and how it caused the malfunction?
Re: What I Wish I Had Known Before Scaling Uber [video]
#220Earlier quoted context omitted.
> I have no idea why people keep thinking microservices is all about scalability. It's an aspect. It's often the beginning of a micro-service migration story in talks I've heard. > Microservices is all about taking a big problem and breaking it down into smaller components ... and putting the network between them. It's all well and good but the tradeoffs are not obvious there either. Most engineers I know who claim t…
You seem to be repeating these weird myths that have no basis in reality. You don't need to be an expert in distributed systems to use microservices. It's literally replacing a function call with an RPC call. That's it. If you want to make tracing easier you tag the user request with an ID and pass it through your API calls or use something like Zipkin. But needing formal verification in order to test your architectu…
No basis at all? I knew I was unhinged...
> You don't need to be an expert in distributed systems to use microservices.
True. Hooray for abstractions. You don't need to understand how the V8 engine allocates and garbage collects memory either... well until you do.
> It's literally replacing a function call with an RPC call.
You're not wrong.
Which is the point. Whether for architectural or performance reasons I think you need to understand your domain and model your data first. For domains that map really well to the microservice architecture you're not going to have many problems.
And a formal specification is overkill for many, many scenarios. That doesn't mean they're useless. They're just not useful, perhaps, for e-commerce sites.
But anywhere you have an RPC call that depends on external state, ordering, consensus... the point is that the tradeoffs are not always immediately apparent unless you know your data really well.
> And I've worked on 2 of the world's top 5 ecommerce sites which both use microservices.
And I've worked on public and private clouds! Cool.
The point was and still is the same whether performance or architecture... think about your data! The rest falls out from that.