Live data from Hacker News

Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

uber.com

211–220 of 233 posts

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#211

Earlier quoted context omitted.

You’ve never been the one neck to choke when things go wrong have you? If Billy Bob’s cloud provider goes down, you are going to constantly be blamed for making a poor decision. If anything goes wrong they are going to question your decision. If you choose AWS (or Azure) and a region goes down - everyone else is down too. “No one ever got fired for choosing IBM”. Choosing the most popular vendor - AWS, Salesforce, Se…

Even if the alternate cloud provider goes offline for an entire day it still would be worth it financially compared to AWS because egress is so expensive there.

I’ve been in places where total AWS spend was a rounding error compared to revenue. Egress fees weren’t a top 10 cost and wasn’t worth optimizing for.

The bosses would’ve blamed me for choosing a tier 2/3 noname provider the first time a day of downtime happens. And they would’ve been right.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#212
post #165

Earlier quoted context omitted.

Do all of those need to be microservices or you could you instead have one monolithic payment service that handled all those use cases?

Of course you could, just like you could do this in 2000 less-well-defined microservices, or 8000 more finely-grained ones. The question is what makes you think 1 service is immediately better than however many payment services there are now?

All other things being equal, 1 service is obviously better than 4,000 services to maintain.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#213

Earlier quoted context omitted.

And when has that happened with respect to either GCP, AWS or Azure at a level that it’s worth migrating? Even if you have done everything in a “cloud agnostic” way, “infrastructure has weight”. Any large migration isn’t just technical , it involves project management, organization training, regression testing, compliance testing, security testing, architecture review boards, vendor negotiations, firewall changes, co…

>And when has that happened with respect to either GCP, AWS or Azure at a level that it’s worth migrating? Egress price makes it worth migrating away from those three.

Egress prices make migrating away hard. But a lot of products don't need to push much data out of AWS, especially with VPC peering based products.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#214

Earlier quoted context omitted.

And why do you think they could answer you with any details without going through comms?

because this is the Internet and anyone can make an anonymous account via VPN, if someone were so inclined.

Yes and I would break my NDA to answer a random question on HN for what personal gain?

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#215
post #95
post #40

Earlier quoted context omitted.

Question for you: how does performance measurement and optimization work in that environment? Is the key some sort of meta tooling that understands relationships between microservices? How would you express such relationships in the first place? Q2: How do you ensure the stubbed deps behave like the real thing? Q3: how do you handle logging and metrics in an unified way across the stack? And related to this: how do y…

> how does performance measurement and optimization work in that environment? SRE here. Generally speaking, each API or each service will have a contract that it must adhere to depending on upstream and downstream relationships and their fail safes. Each service (or API) will then load test in isolation. After that, if you want to be really sure about regressions (which would include fail safes) you load test the who…

> JSON logging with UTC time formatting

perhaps simplest, biggest impact in my log life has been adhering these principles.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#216
I dislike the Uber business itself (horrible treatment of drivers, poor customer service, poor safety controls, bullying of small businesses with Uber Eats, shitty executive level team with questionable ethics).

But the underlying technology which carried them to this point is a fascinating read.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#217

Earlier quoted context omitted.

>And when has that happened with respect to either GCP, AWS or Azure at a level that it’s worth migrating? Egress price makes it worth migrating away from those three.

Egress prices make migrating away hard. But a lot of products don't need to push much data out of AWS, especially with VPC peering based products.

And for things like transferring data to and from S3 within AWS you use an S3 gateway endpoint so data stays within AWS’s network

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#218
post #34

Earlier quoted context omitted.

How else would engineers demonstrate "impact" for promotions? /s

I worked at a SV startup (series A) for a while and an EM once mentioned struggling to keep the number of microservices under the number of engineers.

Is there a compelling article about the ideal microservice to engineer ratio (ie less than 1.0)?

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#219
post #165

Earlier quoted context omitted.

Of course you could, just like you could do this in 2000 less-well-defined microservices, or 8000 more finely-grained ones. The question is what makes you think 1 service is immediately better than however many payment services there are now?

All other things being equal, 1 service is obviously better than 4,000 services to maintain.

Not that obvious, how do you coordinate people from several of teams working on it?

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#220

Earlier quoted context omitted.

None of the layers of abstraction are perfect. You have to deal with the whole mess all the way down. We've had individual EC2 instances go bad where I currently work, with Amazon acknowledging a hardware problem after a ticket is raised. The reality is, quickly resolving the issue means detecting it and moving off of the physical machine. Naturally our tooling has no convenient way to do that, because we have layers…

No the answer is keeping all of your VMs stateless and just using autoscaling with the appropriate health checks. Even if you just having a min/max of 1.

Describe a health check that can detect any possible hardware problem.

The error rate on the machines was higher in both cases, but many requests still succeeded. Amazon certainly didn't detect an issue right away either.

Post reply on HN