Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What I Wish I Had Known Before Scaling Uber [video]

youtube.com

Post reply on HN
Print view View original

191–200 of 284 posts

Re: What I Wish I Had Known Before Scaling Uber [video]

#191

Post by marcosdumay »

threeseed wrote: post #179
agentultra wrote: post #128

What I Wish Small Startups Had Known Before Implementing A Microservices Architecture: Know your data. Are you serving ~1000 requests per second peak and have room to grow? You're not going to gain much efficiency by introducing engineering complexity, latency, and failure modes. Best case scenario and your business performs better than expected... does that mean you have a theoretical upper bound in 100k rps? Still…

I have no idea why people keep thinking microservices is all about scalability. Almost like they've never worked on a problem with them before. Microservices is all about taking a big problem and breaking it down into smaller components, defining the contracts between the components (which an API is), testing the components in isolation and most importantly deploying and running the components independently. It's the…

Because what you described is "encapsulation", and microservices is a very heavyweight and expensive way to get it.

Re: What I Wish I Had Known Before Scaling Uber [video]

#192

Post by jfoutz »

duaneb wrote: post #186
agentultra wrote: post #128

What I Wish Small Startups Had Known Before Implementing A Microservices Architecture: Know your data. Are you serving ~1000 requests per second peak and have room to grow? You're not going to gain much efficiency by introducing engineering complexity, latency, and failure modes. Best case scenario and your business performs better than expected... does that mean you have a theoretical upper bound in 100k rps? Still…

The main benefit of microservices is not performance, it is decoupling concerns. As scale goes up, performance goes down and decoupling gives a better marginal return: it's often easier to work with abstract networking concerns across a system than it is to tease apart often implicit dependenicies in a monolithic deployment. Of course, this is context sensitive and not everyone has easily decouplable code bases, so Y…

I think it's worth noting, it doesn't just split the code. It also splits the teams. 5 engineers will coordinate automagically, they all know what everyone is working on.

At perhaps 15, breaking up into groups of 5 (or whatever) lets small groups build their features without stomping on anyone else's work. It cuts, not only coupling in the code, but coupling in what engineers are talking about.

The two sort of obvious risks, those teams probably should be reshuffled from time to time, so code stays consistent across the organization. If that's done with care, specific projects can get just the right people. If it's done poorly, you sort of wander aimlessly from meaningless project to meaningless project.

The other one is overall performance and reliability. When something fans out to 40 different microservices, tracking down the slow component is a real pain. Figuring out who is best to fix that can be even worse.

Re: What I Wish I Had Known Before Scaling Uber [video]

#193

Post by rhizome »

duaneb wrote: post #186
agentultra wrote: post #128

What I Wish Small Startups Had Known Before Implementing A Microservices Architecture: Know your data. Are you serving ~1000 requests per second peak and have room to grow? You're not going to gain much efficiency by introducing engineering complexity, latency, and failure modes. Best case scenario and your business performs better than expected... does that mean you have a theoretical upper bound in 100k rps? Still…

The main benefit of microservices is not performance, it is decoupling concerns. As scale goes up, performance goes down and decoupling gives a better marginal return: it's often easier to work with abstract networking concerns across a system than it is to tease apart often implicit dependenicies in a monolithic deployment. Of course, this is context sensitive and not everyone has easily decouplable code bases, so Y…

a better marginal return: it's often easier to work with abstract networking concerns across a system than it is to tease apart often implicit dependenicies in a monolithic deployment.

Do the complexities introduced not have a cost, i.e. are those marginal returns offset by the choice in the first place? Call it "platform debt."

Re: What I Wish I Had Known Before Scaling Uber [video]

#194

Post by rhizome »

marcosdumay wrote: post #191
threeseed wrote: post #179

Earlier quoted context omitted.

I have no idea why people keep thinking microservices is all about scalability. Almost like they've never worked on a problem with them before. Microservices is all about taking a big problem and breaking it down into smaller components, defining the contracts between the components (which an API is), testing the components in isolation and most importantly deploying and running the components independently. It's the…

Because what you described is "encapsulation", and microservices is a very heavyweight and expensive way to get it.

I wonder if there's any writing out there about the evolutionary hierarchy of refactoring, like:

code > function/method > module > class > ... > microservice

Re: What I Wish I Had Known Before Scaling Uber [video]

#195

Post by andrewchambers »

yummyfajitas wrote: post #190
threeseed wrote: post #179

Earlier quoted context omitted.

I have no idea why people keep thinking microservices is all about scalability. Almost like they've never worked on a problem with them before. Microservices is all about taking a big problem and breaking it down into smaller components, defining the contracts between the components (which an API is), testing the components in isolation and most importantly deploying and running the components independently. It's the…

You don't need to build a distributed system for that. Just build the ShippingServiceLibrary, let others import the jar file (or whatever) and maintain a stable interface. I wrote about this a couple of years ago in more detail, so I'll just reference that: https://www.chrisstucchio.com/blog/2014/microservices_for_th... HN discussion: https://news.ycombinator.com/item?id=8227721

I think people don't maintain discipline if it isn't forced. So its a round about way of forcing people to maintain boundaries, that wouldn't be needed much of the time if people had stricter development practices.

Re: What I Wish I Had Known Before Scaling Uber [video]

#196

Post by andy_ppp »

ramchip wrote: post #53
andy_ppp wrote: post #32

Earlier quoted context omitted.

I always wonder in these cases about giving each car an actor in Erlang/Elixir and having complete network transparency, message handling and crashes handled for free. The routing is very complex too but as you note scales well, until you want to start routing/pickups based on the realtime location of other cars.

I've given that kind of model a lot of thought too, but I did not find very satisfying solutions to these problems: - How do you distribute the car actors on nodes, assuming the number of nodes is variable? (I think riak_core looks interesting, but it does not seem to have a way to guarantee unicity of something - it's rather built to replicate the data on multiple nodes for redundancy) - What happens if a node fails…

For the netspit stuff I'd look at Phoenix.Presence works and see how that handles it with a CRDT. There are various ways of distributing things across nodes in Erlang/Elixir but maybe you'd need to build something.

I think you are right about the riak core stuff - you could probably keep track of cars using some sort of distributed hash and kill multiple cars if they were to ever spawn.

In fact a way of instantiating processes and finding them based on a CRDT is probably a pretty cool little project...

Re: What I Wish I Had Known Before Scaling Uber [video]

#197

Post by hinkley »

ryandrake wrote: post #159
hinkley wrote: post #153

Earlier quoted context omitted.

So I should fire all my developers is what you're saying... Man, I'm gonna save so much money.

You laugh, but a lot of time the best thing for a software product is to not make so many damn changes. People tend to over-estimate the value of features and tweaks and under-estimate the risk of things going wrong.

Oh I know. I've been in a meeting with several bosses where I tell them flat out "don't give me more developers, give me more hardware and tools". They usually look like I have an extra arm growing out of the top of my head.

Re: What I Wish I Had Known Before Scaling Uber [video]

#198

Post by nichochar »

cagataygurturk wrote: post #96
paukiatwee wrote: post #37

Just to confirm, 1000 microservices in this case is 1000 different apps (e.g.different docker images) running simultaneously? 1000 microservices in this case not 1000 microservice instances (e.g. docker instances)? If it is 1000 microservices as in different apps, then they must have at least 2000 running apps (at least 2 instances per app for HA). Maybe uber only have 200 "active" microservice app running at the sam…

Maybe my ignorance, but do they have services like left-pad-as-a microservice? I can't understand which 1000 microservices you can derive for a cab-renting application.

I used to work in startups, and wondered this all the time. Then I joined a big valley tech company, and now I understand.

It's because they hire smart and ambitious people, but give them a tiny vertical to work on. It's a person level problem, not a technical one in my opinion.

I think you solve this by designing your stack and then hiring meticulously, instead of HIRE ALL THE ENGINEERS and then figure it out (which is quite obviously uber's problem)

Re: What I Wish I Had Known Before Scaling Uber [video]

#199

Post by nichochar »

Honestly I don't think the problem is microservices. I mean everything he brings up is true, but it's more of a "how you do microservices" issue.

I used to work in startups, and overall was impressed with velocity. Then I joined a big valley tech company, and now I understand.

It's because they hire smart and ambitious people, but give them a tiny vertical to work on. On a personal level, you WANT to build something, so you force it.

I think you solve this by designing your stack and then hiring meticulously with rules (like templates for each micro service), instead of HIRE ALL THE ENGINEERS and then figure it out (which is quite obviously uber's problem)

Re: What I Wish I Had Known Before Scaling Uber [video]

#200

Post by andy_ppp »

lgas wrote: post #48
andy_ppp wrote: post #32

Earlier quoted context omitted.

I always wonder in these cases about giving each car an actor in Erlang/Elixir and having complete network transparency, message handling and crashes handled for free. The routing is very complex too but as you note scales well, until you want to start routing/pickups based on the realtime location of other cars.

You're saying you want to extend "just let it crash" to the individual car level? :)

Depends on how smart your supervisor gets doesn't it but sure, why not? Many times processes can be restarted in microseconds...
Post reply on HN