Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adopting Microservices at Netflix: Lessons for Architectural Design

nginx.com

Post reply on HN
Print view View original

31–40 of 76 posts

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#31

Post by danudey »

nawitus wrote: post #16

A few questions: a) How do you prevent technical debt? It seems to be more difficult due to APIs which shouldn't have breaking changes. In theory you could always version up the APIs and serve both versions or just add a new API for a breaking change, but these solutions seems awkward. b) How do you start developing multiple microservices at the same time? I would expect APIs to change a lot in the beginning, which w…

This is basically it. Designing a game? Build the Game service, with all your game logic. Need users and authentication now? Start writing an Identity service, and so on.

The only difference is that instead of starting to write some Identity class and use it in your Game service, you write some Identity class and expose it via a REST API, and then provide an interface library that interfaces with that REST API. Call it IdentityInterface or libidentity or something. Pydentity, whatever. It makes an HTTP request, gets a serialized object, unserializes it, and returns it.

For simplicity, put all your public models in that library, and it gets shared by both the Identity service and the Game service. Those models represent an object and what you can do with it. In the Identity service is where all of that actually happens.

This is also how you solve the 'multiple microservices at the same time' problem; your interface library provides the public interface, and the backend REST API is the 'private' API used by the public interface. You make changes to the backend API and the public library and no one notices, or you make incompatible changes to the public service and fix everything before you deploy; ideally, you add new APIs, migrate services over, then deprecate the old ones.

In the end, each service sees the world as fundamentally the same; there's a library with classes and functionality, and you use that to do things. If you design it right, it's never obvious from your code that you're accessing a different service elsewhere in your infrastructure.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#32

Post by jacques_chester »

mercurial wrote: post #26
jacques_chester wrote: post #20

Earlier quoted context omitted.

> How do you start developing multiple microservices at the same time? Same as any other project: Develop from the outside in. In practice, trying to develop in the "optimal order' leads to speculative development that will be wasted.

Nothing prevents you from having multiple microservices sharing the same codebase. That said, the "it's just like the web" model doesn't sound fantastic to me. It sounds like your app now depends on contracts which are only enforced by good practices, not by something strongly typed you can check at compile time, unless you use something like protocol buffers to generate the boilerplate.

This is where the test-driven world, which in my experience is strongest on the dynamic language side of programming, has come back around full circle.

In microservices, everything is dynamically typed.

There is no single binary produced by a single compiler performing whole-program checks of consistency. Even tools like protobufs don't help when code bases drift, or someone introduces a foreign tool, or someone upgrades versions and introduces a subtle mismatch, or some doesn't know you call their service and shuts it down ...

Turns out that driving from tests, and starting those tests from the outermost consumer, is a fairly well-proved way of coping with such conditions.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#33

Post by jacques_chester »

danudey wrote: post #31
nawitus wrote: post #16

A few questions: a) How do you prevent technical debt? It seems to be more difficult due to APIs which shouldn't have breaking changes. In theory you could always version up the APIs and serve both versions or just add a new API for a breaking change, but these solutions seems awkward. b) How do you start developing multiple microservices at the same time? I would expect APIs to change a lot in the beginning, which w…

This is basically it. Designing a game? Build the Game service, with all your game logic. Need users and authentication now? Start writing an Identity service, and so on. The only difference is that instead of starting to write some Identity class and use it in your Game service, you write some Identity class and expose it via a REST API, and then provide an interface library that interfaces with that REST API. Call…

The problem is that microservices are not objects. They leak reality into your problem domain in a way that simply cannot be made to go away.

If regular object oriented programming languages had method calls that randomly failed, were delayed, sent multiple copies of a response, changed how they behaved without warning, sent half-formed responses ... then yes it would be the same.

Distributed systems are hard, because you cannot change things in two places simultaneously. All synchronisation is limited by the bits you can push down a channel up to, but not exceeding, the speed of light. In a single computer system this problem can be hidden from the programmer. In a distributed system, it cannot.

Probably the most devastating critique of the position that "it's just OO modeling!" came in A Note on Distributed Computing, published in 1994 by Waldo, Wyant, Wollrath and Kendall[0]:

"We look at a number of distributed systems that have attempted to paper over the distinction between local and remote objects, and show that such systems fail to support basic requirements of robustness and reliability. These failures have been masked in the past by the small size of the distributed systems that have been built. In the enterprise-wide distributed systems foreseen in the near future, however, such a masking will be impossible."

[0] http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.7...

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#34

Post by danudey »

mercurial wrote: post #26
jacques_chester wrote: post #20

Earlier quoted context omitted.

> How do you start developing multiple microservices at the same time? Same as any other project: Develop from the outside in. In practice, trying to develop in the "optimal order' leads to speculative development that will be wasted.

Nothing prevents you from having multiple microservices sharing the same codebase. That said, the "it's just like the web" model doesn't sound fantastic to me. It sounds like your app now depends on contracts which are only enforced by good practices, not by something strongly typed you can check at compile time, unless you use something like protocol buffers to generate the boilerplate.

Most systems I've seen these days don't have any compile-time type checking, since they're all written in Ruby, Python, or Node.js.

In the normal case of development, you tend to have a broken-out system. For a game for example:

    + Game code
    --+ User authentication classes/functionality (which accesses DB)
    --+ Messaging classes/functionality (which accesses DB)
    --+ User metrics classes/functionality (which accesses DB)
In the new design you'd have this:

    + Game code
    --+ User authentication classes/functionality (which accesses REST service)
    --+ Messaging classes/functionality (which accesses REST service)
    --+ User metrics classes/functionality (which accesses REST service)
In other words, in a clean design, your Game code is accessing a library which provides user Authentication functionality, one which provides Messaging functionality, and one which provides Metrics functionality.

In this new design, you have exactly the same thing - a library which abstracts the details of communicating with the service, encoding data, etc. A person making changes to those libraries, which other services use, is responsible for either not making backwards-incompatible changes, or, when that isn't possible, working with other teams to ensure a clean upgrade path (or doing it themselves, if your lines are sufficiently blurred).

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#35

Post by chiph »

davidkellis wrote: post #8
shanemhansen wrote: post #7

Earlier quoted context omitted.

Ideally you don't have to sync the data because one service owns that data. Other services request that data via api. In a RESTful world those api requests are cacheable.

But what about the situation where you have an entity service that owns the data for one piece of the domain, for example a People service, and then other services, like the Address service and the Billing service, reference a particular person. In that scenario, I can imagine the Address service and the Billing service would have a foreign key referencing a person in the People service. Then, what happens if the Per…

You could have a service bus, where you publish a "PersonDeleted" message that the other services would subscribe to. It decouples the Person service from all the other related entity services.

You'd have to allow for propagation delay. Plus the possibility of a message storm if you delete something fairly fundamental.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#36

Post by nawitus »

jacques_chester wrote: post #32
mercurial wrote: post #26

Earlier quoted context omitted.

Nothing prevents you from having multiple microservices sharing the same codebase. That said, the "it's just like the web" model doesn't sound fantastic to me. It sounds like your app now depends on contracts which are only enforced by good practices, not by something strongly typed you can check at compile time, unless you use something like protocol buffers to generate the boilerplate.

This is where the test-driven world, which in my experience is strongest on the dynamic language side of programming, has come back around full circle. In microservices, everything is dynamically typed. There is no single binary produced by a single compiler performing whole-program checks of consistency. Even tools like protobufs don't help when code bases drift, or someone introduces a foreign tool, or someone upgr…

This reminded me of the AngularJS team deciding to go with (optional) runtime type checking over compile-time checking (which is what TypeScript has done to JavaScript). Their reasoning was that you can use runtime checking for REST responses, which can be argued to somewhat reduce the need for writing tests.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#37

Post by mercurial »

jacques_chester wrote: post #32
mercurial wrote: post #26

Earlier quoted context omitted.

Nothing prevents you from having multiple microservices sharing the same codebase. That said, the "it's just like the web" model doesn't sound fantastic to me. It sounds like your app now depends on contracts which are only enforced by good practices, not by something strongly typed you can check at compile time, unless you use something like protocol buffers to generate the boilerplate.

This is where the test-driven world, which in my experience is strongest on the dynamic language side of programming, has come back around full circle. In microservices, everything is dynamically typed. There is no single binary produced by a single compiler performing whole-program checks of consistency. Even tools like protobufs don't help when code bases drift, or someone introduces a foreign tool, or someone upgr…

> There is no single binary produced by a single compiler performing whole-program checks of consistency. Even tools like protobufs don't help when code bases drift, or someone introduces a foreign tool, or someone upgrades versions and introduces a subtle mismatch, or some doesn't know you call their service and shuts it down ...

Static typing is not a panacea, but large codebase plus dynamic typing everywhere sounds like a recipe for disaster. No matter the amount of testing.

> Turns out that driving from tests, and starting those tests from the outermost consumer, is a fairly well-proved way of coping with such conditions.

You need tests no matter what. However, static typing means a much greater confidence in your codebase.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#38

Post by nine_k »

jfoutz wrote: post #6

If your app depends on lots of of them, it's only going to run as fast as the slowest dependency. 1% chance of poor performance isn't to bad, the joint distribution of 20 microservices each with a 1% chance, well that gets pretty ugly. In the normal case, everything is great, but the failure modes of each service become a much bigger deal. It's a great architecture, but fan out of dependencies is a real risk.

A low percent of poor performance is much easier to achieve if your service is simple, often so simple its answers can be cached. Even if it's outright down, some requests still can be served from the cache.

If you weave together 20 services to produce one mega-service, it's much harder to optimize for performance and even just keep the implementation correct. Caching of complicated multi-factor answers is less frequently possible.

Also, 20 microservices may feed e.g. 5 large 'end-user' services together, in various combinations. If one of the microservices is slow, only these of the end-user services are affected that actually use it.

Monolithic large services are harder to combine, so the risk that an unrelated remote service somehow gets called in the process and slows things down is higher.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#39

Post by mercurial »

danudey wrote: post #34
mercurial wrote: post #26

Earlier quoted context omitted.

Nothing prevents you from having multiple microservices sharing the same codebase. That said, the "it's just like the web" model doesn't sound fantastic to me. It sounds like your app now depends on contracts which are only enforced by good practices, not by something strongly typed you can check at compile time, unless you use something like protocol buffers to generate the boilerplate.

Most systems I've seen these days don't have any compile-time type checking, since they're all written in Ruby, Python, or Node.js. In the normal case of development, you tend to have a broken-out system. For a game for example: + Game code --+ User authentication classes/functionality (which accesses DB) --+ Messaging classes/functionality (which accesses DB) --+ User metrics classes/functionality (which accesses DB…

> In this new design, you have exactly the same thing - a library which abstracts the details of communicating with the service, encoding data, etc. A person making changes to those libraries, which other services use, is responsible for either not making backwards-incompatible changes, or, when that isn't possible, working with other teams to ensure a clean upgrade path (or doing it themselves, if your lines are sufficiently blurred).

The new design trades a "modular but monolithic" design for complexity and brittleness, IMHO. The ability to spin up new instances of a given service on demand is interesting, but it sure sounds like reinventing Erlang without Erlang's tooling.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#40

Post by mercurial »

chiph wrote: post #35
davidkellis wrote: post #8

Earlier quoted context omitted.

But what about the situation where you have an entity service that owns the data for one piece of the domain, for example a People service, and then other services, like the Address service and the Billing service, reference a particular person. In that scenario, I can imagine the Address service and the Billing service would have a foreign key referencing a person in the People service. Then, what happens if the Per…

You could have a service bus, where you publish a "PersonDeleted" message that the other services would subscribe to. It decouples the Person service from all the other related entity services. You'd have to allow for propagation delay. Plus the possibility of a message storm if you delete something fairly fundamental.

> You could have a service bus, where you publish a "PersonDeleted" message that the other services would subscribe to. It decouples the Person service from all the other related entity services.

You're still screwed if you complete a transaction on the deleted person's still-existing account, now that your system is no longer transactional...

Post reply on HN