Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

uber.com

Post reply on HN
Print view View original

221–230 of 233 posts

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#221

Post by isbvhodnvemrwvn »

herval wrote: post #33
talent_deprived wrote: post #19

Earlier quoted context omitted.

I've seen at least one place with many more than that in recent years. If you have one microservice "listener" per queue and another for the database processing and persistence (business logic) and another providing an API for one or more frontend UI's related to it then the microservice tally goes up very fast. It's kind of surprising to read so many comments indicating HN readers weren't aware of this.

sounds like a massive nightmare

Most are likely limited to some subdomain, with limited communication between domains.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#222

Post by isbvhodnvemrwvn »

dilyevsky wrote: post #158
nvarsj wrote: post #149

Earlier quoted context omitted.

Yup I remember that. Netflix seems to be the poster child for overengineered architecture - for something that is almost entirely commoditised nowadays (one-way over the internet video streaming). Uber's problem space is significantly more complex than Netflix, so I'm unsure it's a fair comparison. But they do seem to have quite a lot of overengineering going on. At least that's how I feel each time I read an Uber te…

> Uber's problem space is significantly more complex than Netflix What makes you say this? Netflix serves probably several orders of magnitude more bytes and online video is hard. At its core Uber is basically a Passenger Service System and we had systems like these implemented in software since 1950s

There are a lot more use cases in taxi and food delivery space than in video streaming. At least by an order of magnitude. Consider various user personas for one, legal considerations and so on. Technically each use case might be less demanding than video streaming, but overall much more complex.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#223

Post by intunderflow »

Oh hey, this is the thing I work on.

We're giving a talk about this at KCD Denmark on the 14th of November "Keynote: Uber - Migrating 2 million CPU cores to Kubernetes" if anyone is in the area and has any particular interest in this.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#224

Post by mooreds »

aitchnyu wrote: post #218
mooreds wrote: post #34

Earlier quoted context omitted.

I worked at a SV startup (series A) for a while and an EM once mentioned struggling to keep the number of microservices under the number of engineers.

Is there a compelling article about the ideal microservice to engineer ratio (ie less than 1.0)?

I don't know of one, but this thread has some interesting discussion: https://www.reddit.com/r/ExperiencedDevs/comments/x1p5gj/my_...

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#225

Post by MichaelZuo »

isbvhodnvemrwvn wrote: post #219
lamontcg wrote: post #212

Earlier quoted context omitted.

All other things being equal, 1 service is obviously better than 4,000 services to maintain.

Not that obvious, how do you coordinate people from several of teams working on it?

IBM managed to coordinate several hundred people on a single software product for decades on end.

Even Microsoft managed to do so for multiple products while also stack ranking the teams.

And I doubt there's a single service, even payments, that's as technically complex as Excel.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#226

Post by s3p »

s3p wrote: post #176
activescott wrote: post #111

> In preparation for the move to the cloud, the company spent two years working towards making all stateless microservices portable so that their placement in zones and regions can be managed centrally without any involvement from the service engineers I'd like to hear more about how Uber organized the engineering teams over two years to make "stateless microservices portable". How many teams? What were the requireme…

Maybe direct these questions to a C-level employee at Uber who could potentially answer them for you?

I'm replying to my own comment here since it was so severely downvoted. OP was musing about a bunch of questions that didn't seem useful to the discussion. Who was he asking? And if you're going to say an Uber staff member, why doesn't his comment indicate that? It just didn't seem to add to the discussion at all.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#227

Post by scarface_74 »

nitwit005 wrote: post #220
scarface_74 wrote: post #206

Earlier quoted context omitted.

No the answer is keeping all of your VMs stateless and just using autoscaling with the appropriate health checks. Even if you just having a min/max of 1.

Describe a health check that can detect any possible hardware problem. The error rate on the machines was higher in both cases, but many requests still succeeded. Amazon certainly didn't detect an issue right away either.

There is no way that you could record metrics - even custom metrics that get populated via the CloudWatch logs agent to CloudWatch and over a certain threshold of errors, bring another instance up and kill the existing instance? If you could detect sporadic errors there must be some method to automated it.

I’m assuming this isn’t a web server, if so it’s even simpler.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#228

Post by lamontcg »

isbvhodnvemrwvn wrote: post #219
lamontcg wrote: post #212

Earlier quoted context omitted.

All other things being equal, 1 service is obviously better than 4,000 services to maintain.

Not that obvious, how do you coordinate people from several of teams working on it?

Do you really have 4,000 teams working on payments?

And I'd agree with the other child comment that the monolith can always be broken into separate components which are owned by different teams.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#229

Post by nitwit005 »

scarface_74 wrote: post #227
nitwit005 wrote: post #220

Earlier quoted context omitted.

Describe a health check that can detect any possible hardware problem. The error rate on the machines was higher in both cases, but many requests still succeeded. Amazon certainly didn't detect an issue right away either.

There is no way that you could record metrics - even custom metrics that get populated via the CloudWatch logs agent to CloudWatch and over a certain threshold of errors, bring another instance up and kill the existing instance? If you could detect sporadic errors there must be some method to automated it. I’m assuming this isn’t a web server, if so it’s even simpler.

A statistical rule moves you into the realm of deciding what rate of false positives and false negatives you'll tolerate. Based on data from exactly two incidents in this case, which is obviously a bit fraught.

Re: Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos

#230

Post by echelon »

scarface_74 wrote: post #214
fragmede wrote: post #210

Earlier quoted context omitted.

because this is the Internet and anyone can make an anonymous account via VPN, if someone were so inclined.

Yes and I would break my NDA to answer a random question on HN for what personal gain?

It's a ridiculously innocuous question. If I were worried about blowback, I'd question working for Uber at all.
Post reply on HN