Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

uber.com

Post reply on HN
Print view View original

161–170 of 233 posts

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#161

Post by folmar »

jgraettinger1 wrote: post #154
scarface_74 wrote: post #148

Earlier quoted context omitted.

And then every time you had a change, you would have to deploy everything and your surface of failure is greater. What would a monolith buy you?

Not having to evolve or understand or staff 4,000 micro services. An ability to easily change the boundaries of your conceptual components, because they WILL be wrong now or in the future.

It might be easier if you have the same API for payment microservices, but each different implementation in a different service, so approximately 100 times less distinct APIs than microservices.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#162

Post by dehrmann »

scarface_74 wrote: post #146

> The team used existing tooling to move services between zones in order to ensure they were portable. Firstly, they allowed services to be moved back to the original zone to resolve any portability issues, but once resolved, services would be moved periodically to validate portability and prevent regressions. This is something that most companies don’t do when they say they want to do $x to “prevent lock in”. Uber a…

It's probably more cost effective to negotiate a long-term max price with your cloud provider with a force majeure clause.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#164

Post by dehrmann »

danpalmer wrote: post #5
x86x87 wrote: post #2

Does Uber really need 4000 microservices?

A different (better?) question is, does Uber need 4000 API contracts? The answer to that is probably yes. APIs let us split work across systems/people/teams/regions, and provide a way for both sides of a split to work together. Uber has a lot of teams, a lot of engineers, and so it makes sense that there are a lot of API boundaries to allow them to work together more efficiently. Sometimes those APIs make sense to pa…

Static function calls are also API contracts.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#165

Post by pc86 »

lamontcg wrote: post #145
jpalomaki wrote: post #138

Earlier quoted context omitted.

There's an an interesting HN comment[1] from 2020 by former Uber engineer, which discusses the complexity a bit. It's more about UI, but the thread discusses the backend as well. In brief something that may look super simple for the user (like handling payments) is actually quite complicated when you cover all the market, different payment types etc. And all this carries to the backend as well. [1] https://news.ycomb…

Do all of those need to be microservices or you could you instead have one monolithic payment service that handled all those use cases?

Of course you could, just like you could do this in 2000 less-well-defined microservices, or 8000 more finely-grained ones.

The question is what makes you think 1 service is immediately better than however many payment services there are now?

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#166

Post by magicalhippo »

scarface_74 wrote: post #150
magicalhippo wrote: post #94

Earlier quoted context omitted.

We're not big on microservices, but we do integrate with a lot of other systems. I find the opaqueness of other services to reduce development speed quite drastically. With local code I can view both sides of the fence and easily see if I'm using it wrong or if it's a bug in my colleagues code. Seems that if you're constantly developing against opaque services you'd end up in the same quagmire quite quickly?

You shouldn’t care about anything outside of published contracts for dependencies. You’re slwags dependent on underlying APIs

Of course I shouldn't.

But when things don't work as I expect, it's far more efficient to be able to view the code on both sides, rather than only on my own side and try to guess what the other side is doing.

Besides the usual suspect of wrong understanding on my end leading to misuse, this can also be due to lacking or wrong documentation of the other system, or bugs in the other system due to unexpected inputs or similar.

Like just a few days ago we spent an unreasonable amount of time with an API of one of our customers, where we would get empty list back for some of our queries. Turned out something in their service crashed when handed national characters, despite accepting JSON and hence UTF-8 input and nothing in the documentation about English letters only. Rather than returning 400 or 500, the service returned 200 with an empty list, leading us to assume we did something wrong.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#167

Post by zo1 »

scarface_74 wrote: post #152
deathanatos wrote: post #140

Earlier quoted context omitted.

There's no way that number isn't fiction; Occam's razor say's its out of the range of believable. That's ~2 per eng according to Google. That's absurd. (That eng headcount is also a bit … high.) This sounds like a figure from someone who sees a signle microservice running across 100 pods/instances, and counted that as 100 "microservices".

S3 alone is built on top of 300 micro services. I don’t find it unbelievable that Uber needs a lot of them.

I find it highly disturbing that so far I've seen Zero Uber devs on this thread adding any sort of context/info or just confirmation. Wtf is this, the NSA? KGB? Can't they just list/dump the names of said 4000 services, or is that somehow some sort of secret-sauce?

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#168

Post by maronato »

davidw wrote: post #18
danpalmer wrote: post #5

Earlier quoted context omitted.

A different (better?) question is, does Uber need 4000 API contracts? The answer to that is probably yes. APIs let us split work across systems/people/teams/regions, and provide a way for both sides of a split to work together. Uber has a lot of teams, a lot of engineers, and so it makes sense that there are a lot of API boundaries to allow them to work together more efficiently. Sometimes those APIs make sense to pa…

I wonder what setting up a local dev instance is like for anything involving more than one or two of those.

When you want to run a local version of a service, you start it locally as normal and configure either local stubs of dependencies and dependents, or you can hook up the actual “production” services to your local service process. If doing the latter, you can create special user accounts and events that change how the networking routing happens. Events from those fake users pass through regular production apps up until the service you’re testing, then they are instead routed to your local version, and you can continue the calls to other prod services.

Uber employees’ apps are special and allow us to log in as these fake users and create fake rides or deliveries, and then we can look at the traces and logs to debug and stuff.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#169

Post by scarface_74 »

magicalhippo wrote: post #166
scarface_74 wrote: post #150

Earlier quoted context omitted.

You shouldn’t care about anything outside of published contracts for dependencies. You’re slwags dependent on underlying APIs

Of course I shouldn't. But when things don't work as I expect, it's far more efficient to be able to view the code on both sides, rather than only on my own side and try to guess what the other side is doing. Besides the usual suspect of wrong understanding on my end leading to misuse, this can also be due to lacking or wrong documentation of the other system, or bugs in the other system due to unexpected inputs or s…

> But when things don't work as I expect, it's far more efficient to be able to view the code on both sides, rather than only on my own side and try to guess what the other side is doing.

Are you able to view the source code of your platform vendor? Everyone is at some level dependent on Black box APIs.

If you can document where with certain input you don’t get the expected output, you reach out to the team that is responsible for it whether internally or externally and they either explain it or they fix it.

This is the API service I’ve been working with over the past five+ years - three actually working at AWS (Professional Services).

https://boto3.amazonaws.com/v1/documentation/api/latest/inde...

I found a bug in one relatively new API that a service team released, I reached out to the team with a documented scenario and they fixed it.

Other times they explained what I was doing wrong. That’s what any large organization does.

I’ve worked with other vendors and internal teams plenty of times over the years.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#170

Post by scarface_74 »

dehrmann wrote: post #162
scarface_74 wrote: post #146

> The team used existing tooling to move services between zones in order to ensure they were portable. Firstly, they allowed services to be moved back to the original zone to resolve any portability issues, but once resolved, services would be moved periodically to validate portability and prevent regressions. This is something that most companies don’t do when they say they want to do $x to “prevent lock in”. Uber a…

It's probably more cost effective to negotiate a long-term max price with your cloud provider with a force majeure clause.

Unless you’re crazy enough to work with GCP, the “my cloud provider is going to lock me in and then raise prices” doesn’t happen. AWS has only raised prices in a few very obscure cases ever.

One of which is putting a price on HEAD requests in S3 (?).

AWS already gives long term price discounts/guaranteed prices for reserve pricing and Big customers already have negotiated contracts.

Post reply on HN