Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

uber.com

Post reply on HN
Print view View original

191–200 of 233 posts

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#191

Post by scarface_74 »

dehrmann wrote: post #184
scarface_74 wrote: post #181

Earlier quoted context omitted.

And when has that happened with respect to either GCP, AWS or Azure at a level that it’s worth migrating? Even if you have done everything in a “cloud agnostic” way, “infrastructure has weight”. Any large migration isn’t just technical , it involves project management, organization training, regression testing, compliance testing, security testing, architecture review boards, vendor negotiations, firewall changes, co…

The other thing I'd add is cloud agnosticism doesn't scale. If everyone were prepared for it, there wouldn't be enough elastic capacity with other cloud providers. You'd need enough reserved capacity in another cloud to pull it off, but I guarantee you finance will say "no." What makes the most sense is multi-region work since it's more cost effective, and it's the more likely failure scenario.

There usually isn’t enough elasticism if a region fails. You really even then need to have reserved capacity and maybe even a hot standby or active-active.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#192

Post by dboreham »

x86x87 wrote: post #40
threeseed wrote: post #22

Earlier quoted context omitted.

I've worked on a couple of extremely large micro services projects. And the thing is that nobody ever needs to run the entire stack other than end to end tests which get run in the cloud. You just checkout the services you need and because they are designed to be isolated the dependencies will usually be automatically stubbed out. So it's just a matter of running them or chaining them together if you have a particula…

Question for you: how does performance measurement and optimization work in that environment? Is the key some sort of meta tooling that understands relationships between microservices? How would you express such relationships in the first place? Q2: How do you ensure the stubbed deps behave like the real thing? Q3: how do you handle logging and metrics in an unified way across the stack? And related to this: how do y…

What you are imagining happens (it's not pretty).

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#193

Post by locustmostest »

I couldn't find any explanation of where the data would be found. Are they splitting data across clouds, and constantly "porting" that data from cloud to cloud as part of their portability?

Orchestrating the application layer across clouds is interesting, but how does their data layer work?

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#194

Post by jvans »

opportune wrote: post #179

I believe the dollar amount savings figures, they’re big and worthy of a congratulations to the engineers involved! IMO, engineering man hour savings are a lot less trustable. This may eliminate or simplify some engineering processes but IME massive migrations like this simply replace them with a different set of processes; because they’re different and theoretically addressable they’re not counted against the hours…

> This may eliminate or simplify some engineering processes but IME massive migrations like this simply replace them with a different set of processes

Yes

> because they’re different

Now I have to learn an entire new set of tools/processes etc that are more useful to someone else but not helpful for me. The old one had its quirks but I knew it inside out and now the whole org has to re-learn how to do everything we did before.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#195

Post by charcircuit »

scarface_74 wrote: post #190
charcircuit wrote: post #186

Earlier quoted context omitted.

>And when has that happened with respect to either GCP, AWS or Azure at a level that it’s worth migrating? Egress price makes it worth migrating away from those three.

You’ve never been the one neck to choke when things go wrong have you? If Billy Bob’s cloud provider goes down, you are going to constantly be blamed for making a poor decision. If anything goes wrong they are going to question your decision. If you choose AWS (or Azure) and a region goes down - everyone else is down too. “No one ever got fired for choosing IBM”. Choosing the most popular vendor - AWS, Salesforce, Se…

Even if the alternate cloud provider goes offline for an entire day it still would be worth it financially compared to AWS because egress is so expensive there.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#196

Post by scarface_74 »

charcircuit wrote: post #195
scarface_74 wrote: post #190

Earlier quoted context omitted.

You’ve never been the one neck to choke when things go wrong have you? If Billy Bob’s cloud provider goes down, you are going to constantly be blamed for making a poor decision. If anything goes wrong they are going to question your decision. If you choose AWS (or Azure) and a region goes down - everyone else is down too. “No one ever got fired for choosing IBM”. Choosing the most popular vendor - AWS, Salesforce, Se…

Even if the alternate cloud provider goes offline for an entire day it still would be worth it financially compared to AWS because egress is so expensive there.

And you ignored the entire reply didn’t you? It’s naive to think at the “one neck to choke level” that all decisions are made for purely technical reasons.

And for you to just say “it’s okay to be down an entire day” because of egress cost tells me that you have never done infrastructure requirements analysis at scale.

First you have to assess the cost of being down for a period of time, then you have to access RTO, RPO requirements and not all workloads have high egress costs - especially things like data lakes that may have a lot of ingress and processing costs, but relatively low egress costs.

I’ve done a lot of different cloud projects over the years from lift and shifts, to data lakes, to cloud call centers, to serverless, to ETL jobs, you can’t just blindly repeat “egress costs” in a vacuum without understanding use cases.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#197

Post by echelon »

s3p wrote: post #176
activescott wrote: post #111

> In preparation for the move to the cloud, the company spent two years working towards making all stateless microservices portable so that their placement in zones and regions can be managed centrally without any involvement from the service engineers I'd like to hear more about how Uber organized the engineering teams over two years to make "stateless microservices portable". How many teams? What were the requireme…

Maybe direct these questions to a C-level employee at Uber who could potentially answer them for you?

There are no doubt lots of Uber employees that post here. This is an appropriate forum to ask.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#198

Post by charcircuit »

scarface_74 wrote: post #196
charcircuit wrote: post #195

Earlier quoted context omitted.

Even if the alternate cloud provider goes offline for an entire day it still would be worth it financially compared to AWS because egress is so expensive there.

And you ignored the entire reply didn’t you? It’s naive to think at the “one neck to choke level” that all decisions are made for purely technical reasons. And for you to just say “it’s okay to be down an entire day” because of egress cost tells me that you have never done infrastructure requirements analysis at scale. First you have to assess the cost of being down for a period of time, then you have to access RTO,…

I never claimed that it is always worth it to switch because of egress costs, but that egress costs are a reason to switch. If I ran my sites on AWS it would 100x the cost of running it.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#199

Post by lokar »

justapassenger wrote: post #106
x86x87 wrote: post #2

Does Uber really need 4000 microservices?

From experience working at big tech I’m willing to take a guess. Maybe a couple of dozens will be actual more complex and meaningful services. Then few dozens more services that are somewhat more unique. And then majority of the long tail will be mostly cookie cutter services, doing X, but for lots of different use cases, where each of use cases is separate deployment counting as a service (for example - systems to p…

The same binary with a different configuration

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#200

Post by scarface_74 »

charcircuit wrote: post #198
scarface_74 wrote: post #196

Earlier quoted context omitted.

And you ignored the entire reply didn’t you? It’s naive to think at the “one neck to choke level” that all decisions are made for purely technical reasons. And for you to just say “it’s okay to be down an entire day” because of egress cost tells me that you have never done infrastructure requirements analysis at scale. First you have to assess the cost of being down for a period of time, then you have to access RTO,…

I never claimed that it is always worth it to switch because of egress costs, but that egress costs are a reason to switch. If I ran my sites on AWS it would 100x the cost of running it.

These were your words

> Egress price makes it worth migrating away from those three.

> Even if the alternate cloud provider goes offline for an entire day it still would be worth it financially compared to AWS because egress is so expensive there.

You never qualified either with “in my particular use case”. If you had, I would have had no argument. I haven’t been flown into your company along with SAs, sales, project managers, etc for a week to do a proper “as-is” assessment and to see what your requirements are.

I haven’t accessed the competencies of your staff or determined what is your competitive advantage and what is the “undifferentiated heavy lifting” in your company.

I would never make any blanket statements without knowing your specific use case and automatically assume “cloud” is always the right or wrong answer

Post reply on HN