Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

uber.com

Post reply on HN
Print view View original

171–180 of 233 posts

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#171

Post by folmar »

mardifoufs wrote: post #115
this_user wrote: post #41

For a company that is basically a taxi service, they seem to invest an awful lot in constant rebuilds of their extremely complex infrastructure, which raises the question of whether that is even remotely necessary or just an exercise in pretending that they are a tech company.

They do food delivery, parcel courriers, regular ubers, plan ahead uber, grocery shopping, and a lot of other stuff. if anything this is simpler than most silo driven architectures you'd usually get with such a massively diversified business.

To be fair all of those listed were also handled by taxis previously, just the process was more manual and more distributed, the dispatcher allocated a cab to the requester and maybe passed an initial message, and then it was directly between you and the driver.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#172

Post by scarface_74 »

zo1 wrote: post #167
scarface_74 wrote: post #152

Earlier quoted context omitted.

S3 alone is built on top of 300 micro services. I don’t find it unbelievable that Uber needs a lot of them.

I find it highly disturbing that so far I've seen Zero Uber devs on this thread adding any sort of context/info or just confirmation. Wtf is this, the NSA? KGB? Can't they just list/dump the names of said 4000 services, or is that somehow some sort of secret-sauce?

A random engineer at any company is not going to divulged non public information about the inner working of their company without permission.

We had to sign something at AWS not to divulge internal tooling like what we used for our internal account factory that we used to create AWS accounts. Literally tens of thousands of people know what this tool is.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#174

Post by x86x87 »

scarface_74 wrote: post #172
zo1 wrote: post #167

Earlier quoted context omitted.

I find it highly disturbing that so far I've seen Zero Uber devs on this thread adding any sort of context/info or just confirmation. Wtf is this, the NSA? KGB? Can't they just list/dump the names of said 4000 services, or is that somehow some sort of secret-sauce?

A random engineer at any company is not going to divulged non public information about the inner working of their company without permission. We had to sign something at AWS not to divulge internal tooling like what we used for our internal account factory that we used to create AWS accounts. Literally tens of thousands of people know what this tool is.

you see someone telling you how many microservices s3 has right above.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#175

Post by scarface_74 »

x86x87 wrote: post #174
scarface_74 wrote: post #172

Earlier quoted context omitted.

A random engineer at any company is not going to divulged non public information about the inner working of their company without permission. We had to sign something at AWS not to divulge internal tooling like what we used for our internal account factory that we used to create AWS accounts. Literally tens of thousands of people know what this tool is.

you see someone telling you how many microservices s3 has right above.

Yes that “someone” was me - a former employee who couldn’t remember whether that information was public or whether it was something I was exposed to from the inside.

It in fact was public.

https://aws.amazon.com/blogs/storage/how-automated-reasoning....

I verified that before I posted that little tidbit.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#176

Post by s3p »

activescott wrote: post #111

> In preparation for the move to the cloud, the company spent two years working towards making all stateless microservices portable so that their placement in zones and regions can be managed centrally without any involvement from the service engineers I'd like to hear more about how Uber organized the engineering teams over two years to make "stateless microservices portable". How many teams? What were the requireme…

Maybe direct these questions to a C-level employee at Uber who could potentially answer them for you?

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#177

Post by s3p »

lamontcg wrote: post #145
jpalomaki wrote: post #138

Earlier quoted context omitted.

There's an an interesting HN comment[1] from 2020 by former Uber engineer, which discusses the complexity a bit. It's more about UI, but the thread discusses the backend as well. In brief something that may look super simple for the user (like handling payments) is actually quite complicated when you cover all the market, different payment types etc. And all this carries to the backend as well. [1] https://news.ycomb…

Do all of those need to be microservices or you could you instead have one monolithic payment service that handled all those use cases?

And what service would that be?

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#178

Post by nitwit005 »

jbotdev wrote: post #23

It seems like they’ve gotten to the “holy grail” of deployment where developers don’t have to worry about infrastructure at all in theory. I’ve seen many teams go for simple/leaky abstractions on top Kubernetes to provide a similar solution, which is tempting because it’s easy and flexible. The problem is then all your devs need to be trained in all the complexities of Kubernetes deployments anyway. Hopefully Uber ab…

None of the layers of abstraction are perfect. You have to deal with the whole mess all the way down.

We've had individual EC2 instances go bad where I currently work, with Amazon acknowledging a hardware problem after a ticket is raised. The reality is, quickly resolving the issue means detecting it and moving off of the physical machine.

Naturally our tooling has no convenient way to do that, because we have layers of things trying to pretend physical machines don't matter.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#179

Post by opportune »

I believe the dollar amount savings figures, they’re big and worthy of a congratulations to the engineers involved!

IMO, engineering man hour savings are a lot less trustable. This may eliminate or simplify some engineering processes but IME massive migrations like this simply replace them with a different set of processes; because they’re different and theoretically addressable they’re not counted against the hours saved as they can be bucketed into bugs/to be addressed by the roadmap/legacy behavior migrated from the old system (which is now dangerously-fragile-legacy and not ol-reliable-legac). Eventually someone will come along and decide this too is an inherently flawed platform that needs to be entirely replaced at great expense, and the circle of life continues.

This is still a massive undertaking not just from an engineering perspective but from an organizational/process one though. Whoever pulled this off essentially had to coordinate (or figure out how to simplify/explain things well enough to skip coordination) with almost every engineer and likely almost every production service in a company with thousands of engineers. Those in startups may balk about this kind of thing taking two years, but having done my own two year projects (at a smaller but comparable scale) in a big company I can say two years is what I’d consider a highly optimistic and unlikely outcome for a project of this magnitude.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#180

Post by Narkov »

scarface_74 wrote: post #170
dehrmann wrote: post #162

Earlier quoted context omitted.

It's probably more cost effective to negotiate a long-term max price with your cloud provider with a force majeure clause.

Unless you’re crazy enough to work with GCP, the “my cloud provider is going to lock me in and then raise prices” doesn’t happen. AWS has only raised prices in a few very obscure cases ever. One of which is putting a price on HEAD requests in S3 (?). AWS already gives long term price discounts/guaranteed prices for reserve pricing and Big customers already have negotiated contracts.

Price increases are just one way you can get screwed. You can also lose out when your provider doesn't drop prices or pick up operating efficiencies that other providers have.
Post reply on HN