Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

uber.com

Post reply on HN
Print view View original

181–190 of 233 posts

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#181

Post by scarface_74 »

Narkov wrote: post #180
scarface_74 wrote: post #170

Earlier quoted context omitted.

Unless you’re crazy enough to work with GCP, the “my cloud provider is going to lock me in and then raise prices” doesn’t happen. AWS has only raised prices in a few very obscure cases ever. One of which is putting a price on HEAD requests in S3 (?). AWS already gives long term price discounts/guaranteed prices for reserve pricing and Big customers already have negotiated contracts.

Price increases are just one way you can get screwed. You can also lose out when your provider doesn't drop prices or pick up operating efficiencies that other providers have.

And when has that happened with respect to either GCP, AWS or Azure at a level that it’s worth migrating?

Even if you have done everything in a “cloud agnostic” way, “infrastructure has weight”. Any large migration isn’t just technical , it involves project management, organization training, regression testing, compliance testing, security testing, architecture review boards, vendor negotiations, firewall changes, coordination with third parties who may only allow list certain IP addresses, data migration, etc.

Heck they often have multiple physical network connections to the cloud provider (Direct Connect)

Anyone who thinks they can run everything on K8s and they have “cloud agnosticism” has never done a very large scale migration.

You would be amazed how long it takes to do a bog standard lift and shift of a hundreds of plain jane VMs and VM hosted databases. You can’t get anymore cloud agnostic than that.

source: I’ve done a few over the years in both the “real world” and working in the cloud consulting department at AWS (Professional Services). I no longer work at AWS and have no specific loyalty to AWS.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#182

Post by opportune »

zeroCalories wrote: post #38
jbotdev wrote: post #23

It seems like they’ve gotten to the “holy grail” of deployment where developers don’t have to worry about infrastructure at all in theory. I’ve seen many teams go for simple/leaky abstractions on top Kubernetes to provide a similar solution, which is tempting because it’s easy and flexible. The problem is then all your devs need to be trained in all the complexities of Kubernetes deployments anyway. Hopefully Uber ab…

It's not clear to me that being completely unaware of your infrastructure is a good thing. I don't think it's too much trouble to ask an engineer to understand k8s and think about where their service will live, even if it's a ci system that actually deploys. Furthermore, many layers of abstraction, especially in-house abstraction, just mean you have more code to maintain, another system for people to learn, and exist…

There is a wide spectrum of infrastructure (and platforms, frameworks, etc) from “allows applications to do just about anything, though it may be very complex” to “severely constrains applications but greatly simplifies doing things within those constraints.” To be clear by “just about anything” I am not talking about whether some business logic is expressable, but whether you can eg use EBPF and cgroups, use some esoteric network protocol, run a stateful service that pulls from a queue, issue any network call to anything on the Internet, etc.

If you are developing applications software like Uber 99.99% of the time you really do not need to be doing anything “fancy” or “exotic” in your service. Your service receives data, does some stuff with it (connects to a db or issues calls to other services), returns data. If you let those 0.01% of the things dictate where your internal platform falls on that spectrum, you will make things much more complicated and difficult for 99.99% of the other stuff. Those are where leaky abstractions and bugs come from, both from the platform trying to be more general than it needs to be and from pushing poorly understand boilerplate tasks (like configuring auth, certifications, TLS manually for each service) to infrastructure users.

Being unaware (of course not completely unaware, but essentially not needing to actively consider it while doing things) of infrastructure is actually the ideal state, provided that lack of awareness is because “it just works so well it doesn’t need to be considered”. It means that it lets people get shit done without pushing configuration and leaky abstractions onto them.

I’ll give you one example of something that does an excellent job of this: Linux. Application memory in linux requires some very complex work under the hood, but it has decent default configurations with only a couple commonly changed parameters that most applications don’t need much, and it had a very simple API for applications to interface with. Similar with send/receive syscalls and the use of files for I/O ranging from remote networking to IPC to local disk. These are wonderful APIs and abstractions that simplify very hard problems. The problem with in-house abstraction isn’t that they are trying to do abstractions but that sometimes they just don’t do a good job or churn through them faster than it takes them to stabilize.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#183

Post by dehrmann »

Narkov wrote: post #180
scarface_74 wrote: post #170

Earlier quoted context omitted.

Unless you’re crazy enough to work with GCP, the “my cloud provider is going to lock me in and then raise prices” doesn’t happen. AWS has only raised prices in a few very obscure cases ever. One of which is putting a price on HEAD requests in S3 (?). AWS already gives long term price discounts/guaranteed prices for reserve pricing and Big customers already have negotiated contracts.

Price increases are just one way you can get screwed. You can also lose out when your provider doesn't drop prices or pick up operating efficiencies that other providers have.

That's a long-term concern, though. The thing to worry about is a rug-pull, but no major provider will do that. They could, but they won't.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#184

Post by dehrmann »

scarface_74 wrote: post #181
Narkov wrote: post #180

Earlier quoted context omitted.

Price increases are just one way you can get screwed. You can also lose out when your provider doesn't drop prices or pick up operating efficiencies that other providers have.

And when has that happened with respect to either GCP, AWS or Azure at a level that it’s worth migrating? Even if you have done everything in a “cloud agnostic” way, “infrastructure has weight”. Any large migration isn’t just technical , it involves project management, organization training, regression testing, compliance testing, security testing, architecture review boards, vendor negotiations, firewall changes, co…

The other thing I'd add is cloud agnosticism doesn't scale. If everyone were prepared for it, there wouldn't be enough elastic capacity with other cloud providers. You'd need enough reserved capacity in another cloud to pull it off, but I guarantee you finance will say "no." What makes the most sense is multi-region work since it's more cost effective, and it's the more likely failure scenario.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#185

Post by gen220 »

x86x87 wrote: post #40
threeseed wrote: post #22

Earlier quoted context omitted.

I've worked on a couple of extremely large micro services projects. And the thing is that nobody ever needs to run the entire stack other than end to end tests which get run in the cloud. You just checkout the services you need and because they are designed to be isolated the dependencies will usually be automatically stubbed out. So it's just a matter of running them or chaining them together if you have a particula…

Question for you: how does performance measurement and optimization work in that environment? Is the key some sort of meta tooling that understands relationships between microservices? How would you express such relationships in the first place? Q2: How do you ensure the stubbed deps behave like the real thing? Q3: how do you handle logging and metrics in an unified way across the stack? And related to this: how do y…

I've worked at a place that architected 100s of microservices pretty well, in a similar way that Uber apparently does.

Q1: (perf) these tools exist, the buzzword phrase is "distributed tracing". The relationships are actually not explicitly defined for the tooling to work, but rather inferred. Visualize a network call as a call-stack, where each service is a level in the stack. Jaeger (a CNCF project addressing distributed tracing) was coincidentally started by Uber.

Q2 (stubs): In my experience, mocked responses get you a long, long way. Typically the API response type that you're mocking is generated from a protobuf (or thrift, OpenAPI, etc.) file. If your dependency changes that type in a way that breaks your test, the CI platform will let them know.

If it's a more subtle change (like, it used to deterministically return 18 and now it deterministically returns 20), it's really on the service owners to communicate changes and grep the code base before making the change.

Q3 (logging/metrics): Typically by using shared "logging" and "metrics" lib for each language. Every service will typically be a gRPC service and accordingly a standardized + generated-from-protobufs set of metrics to Prometheus, by default.

Q4 (how to upgrade common libraries): this is definitely a tricky one. The answer is, basically, really carefully. Typically, you'll want your infrastructure to be compatible with vX and vX+1, and give teams a deadline to cut over from logging X to X+1. The couple of weeks before that deadline usually involves a lot of cat-herding and handwringing.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#186

Post by charcircuit »

scarface_74 wrote: post #181
Narkov wrote: post #180

Earlier quoted context omitted.

Price increases are just one way you can get screwed. You can also lose out when your provider doesn't drop prices or pick up operating efficiencies that other providers have.

And when has that happened with respect to either GCP, AWS or Azure at a level that it’s worth migrating? Even if you have done everything in a “cloud agnostic” way, “infrastructure has weight”. Any large migration isn’t just technical , it involves project management, organization training, regression testing, compliance testing, security testing, architecture review boards, vendor negotiations, firewall changes, co…

>And when has that happened with respect to either GCP, AWS or Azure at a level that it’s worth migrating?

Egress price makes it worth migrating away from those three.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#187

Post by gen220 »

magicalhippo wrote: post #94
jameshart wrote: post #78

Earlier quoted context omitted.

In general in a micro service environment, you try to build things so that 1) you don't need to run other things locally, and 2) if you did need to, the services are just containers so it's pretty easy to run one. But you tend to try to write your service so that it treats everything else it depends on like a vendor-provided API. Like, if you were building a Slack bot, you wouldn't ask Slack to let you pull down and…

We're not big on microservices, but we do integrate with a lot of other systems. I find the opaqueness of other services to reduce development speed quite drastically. With local code I can view both sides of the fence and easily see if I'm using it wrong or if it's a bug in my colleagues code. Seems that if you're constantly developing against opaque services you'd end up in the same quagmire quite quickly?

This is exactly why the "microservices" pattern is usually adopted along with the "monorepo" pattern. IMO, it's a strong anti-pattern to have the former without the latter.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#188

Post by intelVISA »

charcircuit wrote: post #186
scarface_74 wrote: post #181

Earlier quoted context omitted.

And when has that happened with respect to either GCP, AWS or Azure at a level that it’s worth migrating? Even if you have done everything in a “cloud agnostic” way, “infrastructure has weight”. Any large migration isn’t just technical , it involves project management, organization training, regression testing, compliance testing, security testing, architecture review boards, vendor negotiations, firewall changes, co…

>And when has that happened with respect to either GCP, AWS or Azure at a level that it’s worth migrating? Egress price makes it worth migrating away from those three.

Whilst there are some nice toys e.g. Spanner, there's generally little other reason to tolerate the abusive pricing of 'the cloud' ime

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#189

Post by intelVISA »

opportune wrote: post #182
zeroCalories wrote: post #38

Earlier quoted context omitted.

It's not clear to me that being completely unaware of your infrastructure is a good thing. I don't think it's too much trouble to ask an engineer to understand k8s and think about where their service will live, even if it's a ci system that actually deploys. Furthermore, many layers of abstraction, especially in-house abstraction, just mean you have more code to maintain, another system for people to learn, and exist…

There is a wide spectrum of infrastructure (and platforms, frameworks, etc) from “allows applications to do just about anything, though it may be very complex” to “severely constrains applications but greatly simplifies doing things within those constraints.” To be clear by “just about anything” I am not talking about whether some business logic is expressable, but whether you can eg use EBPF and cgroups, use some es…

Well put, 99% of companies don't need to introduce such complexity for their relatively trivial use cases (though well-intentioned albeit bad engineers will try to invent it anyway).

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#190

Post by scarface_74 »

charcircuit wrote: post #186
scarface_74 wrote: post #181

Earlier quoted context omitted.

And when has that happened with respect to either GCP, AWS or Azure at a level that it’s worth migrating? Even if you have done everything in a “cloud agnostic” way, “infrastructure has weight”. Any large migration isn’t just technical , it involves project management, organization training, regression testing, compliance testing, security testing, architecture review boards, vendor negotiations, firewall changes, co…

>And when has that happened with respect to either GCP, AWS or Azure at a level that it’s worth migrating? Egress price makes it worth migrating away from those three.

You’ve never been the one neck to choke when things go wrong have you? If Billy Bob’s cloud provider goes down, you are going to constantly be blamed for making a poor decision. If anything goes wrong they are going to question your decision.

If you choose AWS (or Azure) and a region goes down - everyone else is down too. “No one ever got fired for choosing IBM”.

Choosing the most popular vendor - AWS, Salesforce, ServiceNow, or whatever vendor is in the upper right Gartner magic square quadrant never gets questioned by the powers that be.

Post reply on HN