Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adopting Microservices at Netflix: Lessons for Architectural Design

nginx.com

Post reply on HN
Print view View original

51–60 of 76 posts

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#51

Post by threeseed »

mercurial wrote: post #40
chiph wrote: post #35

Earlier quoted context omitted.

You could have a service bus, where you publish a "PersonDeleted" message that the other services would subscribe to. It decouples the Person service from all the other related entity services. You'd have to allow for propagation delay. Plus the possibility of a message storm if you delete something fairly fundamental.

> You could have a service bus, where you publish a "PersonDeleted" message that the other services would subscribe to. It decouples the Person service from all the other related entity services. You're still screwed if you complete a transaction on the deleted person's still-existing account, now that your system is no longer transactional...

You can setup distributed transactions e.g. using Zookeeper, Consul, Redis etc.

It adds complexity of course but that is the give/take when you use microservices.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#52

Post by pmahoney »

danudey wrote: post #29
jfoutz wrote: post #6

If your app depends on lots of of them, it's only going to run as fast as the slowest dependency. 1% chance of poor performance isn't to bad, the joint distribution of 20 microservices each with a 1% chance, well that gets pretty ugly. In the normal case, everything is great, but the failure modes of each service become a much bigger deal. It's a great architecture, but fan out of dependencies is a real risk.

This is solved because you can scale each service differently. Too many DB queries from your messaging service? Upgrade the DB, add another read slave. Too much CPU load on your image processing service? Add more image processing nodes. Breaking your system out into multiple dependencies means you can not only scale your infrastructure, but you can scale individual parts of your infrastructure based on demand, bottle…

> Rating service having problems? Don't show user ratings. Search service offline for updates? Disable search

As an example of a (probably?) bad way to organize services, I worked on a project that had factored a role-based access control system into its own service. Every single web request hit this service, which made it a single point of failure, performance critical, impossible to temporarily disable, etc.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#53

Post by Xorlev »

threeseed wrote: post #50
shanemhansen wrote: post #43

Earlier quoted context omitted.

So the problem you've identified is real. I used to have some bootleg footage of some private amazon tech talks where the speaker emphasized that in distributed systems it was generally a terrible idea to have transactions span entities. I think you basically have to learn to live in an eventually consistent world. In the case of people being deleted I would imagine that the user service exposes a pub/sub interface w…

You don't HAVE to live in an eventually consistent world. If you use something like ZeroMQ or use REST then you can "notify" other services of a "person deleted" event in a synchronous manner.

That assumes the network is always good and services are up. Welcome back to eventual consistency (or none at all)

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#54

Post by shanemhansen »

pmahoney wrote: post #52
danudey wrote: post #29

Earlier quoted context omitted.

This is solved because you can scale each service differently. Too many DB queries from your messaging service? Upgrade the DB, add another read slave. Too much CPU load on your image processing service? Add more image processing nodes. Breaking your system out into multiple dependencies means you can not only scale your infrastructure, but you can scale individual parts of your infrastructure based on demand, bottle…

> Rating service having problems? Don't show user ratings. Search service offline for updates? Disable search As an example of a (probably?) bad way to organize services, I worked on a project that had factored a role-based access control system into its own service. Every single web request hit this service, which made it a single point of failure, performance critical, impossible to temporarily disable, etc.

One alternative to centralized role servers is to use client certificates. I've used x509 certs for this purpose. They are pretty hairy, but so is rolling your own authentication/authorization/token system.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#55

Post by brown9-2 »

exacube wrote: post #15

This seems like it's just taking the idea of decoupling your service and talking to them via APIs a little further by saying.. decouple them to an even smaller granularity? This has always been the generally accepted way to scale out software services. Is there a novel idea being discussed here, or just that they've been doing this at Netflix?

Yes, a lot of organizations already design their backends in this way without necessarily thinking to use a brand new term to describe it.

Cockcroft defines a microservices architecture as a service-oriented architecture composed of loosely coupled elements that have bounded contexts.

To me this just reads as "service-oriented architecture in a sensible way".

No one thinks to intentionally build a SOA with tightly coupled components with poor boundaries.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#56

Post by threeseed »

Xorlev wrote: post #53
threeseed wrote: post #50

Earlier quoted context omitted.

You don't HAVE to live in an eventually consistent world. If you use something like ZeroMQ or use REST then you can "notify" other services of a "person deleted" event in a synchronous manner.

That assumes the network is always good and services are up. Welcome back to eventual consistency (or none at all)

If the network is bad then your monolithic app wouldn't work either.

The problem of services being up/down has been solved with service discovery e.g. Consul, Etcd, Zookeeper.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#57

Post by polynomeal »

My team recently added a microservice to support our fairly monolythic backend service. The big challenge we found was that it takes a lot of effort to make a new (micro)service. We need to create a system of alarms (instead of relying on existing catch all defaults). We need its own test environment, we need to find ways to send traffic to pre-prod. We needed to figure out how to bootstrap the new service into the companies infrastructure. We needed to think about how the dependent service would authenticate against the new service. All of that on top of the core feature work.

All these things are good. You want isolated, focused test environments. You want tightly defined alarms. However we underestimated how long creating a new service would take. In the end we ended up pushing features out when they were ready but before the operational work was complete. Unsurprisingly we saw the issues we knew we wanted to protect against.

Better microservice franeworks that match the companies infrastructure would be helpful. Make building microservices cheap by building tools to speed up the process.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#58

Post by vzaneric »

jacques_chester wrote: post #20
nawitus wrote: post #16

A few questions: a) How do you prevent technical debt? It seems to be more difficult due to APIs which shouldn't have breaking changes. In theory you could always version up the APIs and serve both versions or just add a new API for a breaking change, but these solutions seems awkward. b) How do you start developing multiple microservices at the same time? I would expect APIs to change a lot in the beginning, which w…

> How do you start developing multiple microservices at the same time? Same as any other project: Develop from the outside in. In practice, trying to develop in the "optimal order' leads to speculative development that will be wasted.

http://en.wikipedia.org/wiki/Outside%E2%80%93in_software_dev...

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#59

Post by saryant »

threeseed wrote: post #56
Xorlev wrote: post #53

Earlier quoted context omitted.

That assumes the network is always good and services are up. Welcome back to eventual consistency (or none at all)

If the network is bad then your monolithic app wouldn't work either. The problem of services being up/down has been solved with service discovery e.g. Consul, Etcd, Zookeeper.

That has nothing to do with the fact that if your systems are distributed, you will have eventual consistency.

If System A needs to tell System B about an event in order for A and B to remain consistent, but B is down, you've got eventual consistency, because B can't become consistent with A until it's back up and has performed whatever recovery is necessary to process that event. Service discovery does nothing to solve that problem.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#60

Post by cam- »

polynomeal wrote: post #57

My team recently added a microservice to support our fairly monolythic backend service. The big challenge we found was that it takes a lot of effort to make a new (micro)service. We need to create a system of alarms (instead of relying on existing catch all defaults). We need its own test environment, we need to find ways to send traffic to pre-prod. We needed to figure out how to bootstrap the new service into the c…

Front End teams tend not like micro services, there is too much overhead in getting too little data. As an example we integrate with one micro service where get back a boolean and and a date. We have the overhead of an http call and all the error handling that goes with it for two pieces of data which would be better aggregated into another service. We story point an integration with a new service as an 8, but adding a new field (or two) in an existing API data structure is a 1.

I hope micro services is not just a new fashion in software and is actually useful ten years from now.

Post reply on HN