Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adopting Microservices at Netflix: Lessons for Architectural Design

nginx.com

Post reply on HN
Print view View original

41–50 of 76 posts

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#41

Post by jacques_chester »

mercurial wrote: post #37
jacques_chester wrote: post #32

Earlier quoted context omitted.

This is where the test-driven world, which in my experience is strongest on the dynamic language side of programming, has come back around full circle. In microservices, everything is dynamically typed. There is no single binary produced by a single compiler performing whole-program checks of consistency. Even tools like protobufs don't help when code bases drift, or someone introduces a foreign tool, or someone upgr…

> There is no single binary produced by a single compiler performing whole-program checks of consistency. Even tools like protobufs don't help when code bases drift, or someone introduces a foreign tool, or someone upgrades versions and introduces a subtle mismatch, or some doesn't know you call their service and shuts it down ... Static typing is not a panacea, but large codebase plus dynamic typing everywhere sound…

As soon as you distribute your system, you have dynamic typing, whether you like it or not.

At runtime you are inspecting incoming messages and then routing them to code. It doesn't matter what language the code is written in, it will need to route and validate the messages at runtime.

The type system cannot provide compile-time assurances of behaviour, because it cannot create a single consistent binary which enforces the guarantees.

Your only remaining tool is to drive code from tests and only from tests.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#42

Post by MichaelGG »

exacube wrote: post #15

This seems like it's just taking the idea of decoupling your service and talking to them via APIs a little further by saying.. decouple them to an even smaller granularity? This has always been the generally accepted way to scale out software services. Is there a novel idea being discussed here, or just that they've been doing this at Netflix?

Exactly. Even in the late 90s "3 tier" development pushed the idea of having a separate layer and sticking to APIs. This is just saying that instead of making big layers, make smaller ones. Which isn't bad, it's just that most of the time you don't need this. Add a new service, like "GetRecommendationFromFacebook?" OK, go ahead and deploy, because you haven't changed the rest of the services in your layer, like "LoginUser".

And there was a whole hubbub of discovery what with UDDI and DISCO and all that jazz I never really understood.

It'd be nice if they gave some solid examples of the "micro" part to distinguish it from the general SOA idea.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#43

Post by shanemhansen »

davidkellis wrote: post #8
shanemhansen wrote: post #7

Earlier quoted context omitted.

Ideally you don't have to sync the data because one service owns that data. Other services request that data via api. In a RESTful world those api requests are cacheable.

But what about the situation where you have an entity service that owns the data for one piece of the domain, for example a People service, and then other services, like the Address service and the Billing service, reference a particular person. In that scenario, I can imagine the Address service and the Billing service would have a foreign key referencing a person in the People service. Then, what happens if the Per…

So the problem you've identified is real. I used to have some bootleg footage of some private amazon tech talks where the speaker emphasized that in distributed systems it was generally a terrible idea to have transactions span entities.

I think you basically have to learn to live in an eventually consistent world. In the case of people being deleted I would imagine that the user service exposes a pub/sub interface where address and billing services subscribe to "delete" events.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#44

Post by mercurial »

jacques_chester wrote: post #41
mercurial wrote: post #37

Earlier quoted context omitted.

> There is no single binary produced by a single compiler performing whole-program checks of consistency. Even tools like protobufs don't help when code bases drift, or someone introduces a foreign tool, or someone upgrades versions and introduces a subtle mismatch, or some doesn't know you call their service and shuts it down ... Static typing is not a panacea, but large codebase plus dynamic typing everywhere sound…

As soon as you distribute your system, you have dynamic typing, whether you like it or not. At runtime you are inspecting incoming messages and then routing them to code. It doesn't matter what language the code is written in, it will need to route and validate the messages at runtime. The type system cannot provide compile-time assurances of behaviour, because it cannot create a single consistent binary which enforc…

> As soon as you distribute your system, you have dynamic typing, whether you like it or not.

You have serialization/deserialization issues. You can still type your messages.

> At runtime you are inspecting incoming messages and then routing them to code. It doesn't matter what language the code is written in, it will need to route and validate the messages at runtime.

Of course.

> The type system cannot provide compile-time assurances of behaviour, because it cannot create a single consistent binary which enforces the guarantees.

If you make the assumption that you deploy up-to-date binaries, then knowing at compile time that your producer and consumer use the same data structure for the messages they exchange would give me much better confidence than "it looks like the API conforms to what's written on the wiki".

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#45

Post by jacques_chester »

mercurial wrote: post #44
jacques_chester wrote: post #41

Earlier quoted context omitted.

As soon as you distribute your system, you have dynamic typing, whether you like it or not. At runtime you are inspecting incoming messages and then routing them to code. It doesn't matter what language the code is written in, it will need to route and validate the messages at runtime. The type system cannot provide compile-time assurances of behaviour, because it cannot create a single consistent binary which enforc…

> As soon as you distribute your system, you have dynamic typing, whether you like it or not. You have serialization/deserialization issues. You can still type your messages. > At runtime you are inspecting incoming messages and then routing them to code. It doesn't matter what language the code is written in, it will need to route and validate the messages at runtime. Of course. > The type system cannot provide comp…

> You can still type your messages.

You can hope that they respect the type. For a robust distributed system, you will have to check everything at runtime.

> If you make the assumption that you deploy up-to-date binaries, then knowing at compile time that your producer and consumer use the same data structure for the messages they exchange would give me much better confidence than "it looks like the API conforms to what's written on the wiki".

My reading is that we agree that running code is the only source of truth, we disagree on what guarantees distribution deprives us of.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#46

Post by newobj »

shanemhansen wrote: post #43
davidkellis wrote: post #8

Earlier quoted context omitted.

But what about the situation where you have an entity service that owns the data for one piece of the domain, for example a People service, and then other services, like the Address service and the Billing service, reference a particular person. In that scenario, I can imagine the Address service and the Billing service would have a foreign key referencing a person in the People service. Then, what happens if the Per…

So the problem you've identified is real. I used to have some bootleg footage of some private amazon tech talks where the speaker emphasized that in distributed systems it was generally a terrible idea to have transactions span entities. I think you basically have to learn to live in an eventually consistent world. In the case of people being deleted I would imagine that the user service exposes a pub/sub interface w…

Hardly need "private bootleg" footage to discover this reality. Pat Helland (at the time, working at Amazon) wrote a paper about it maybe 10 years ago.

http://adrianmarriott.net/logosroot/papers/LifeBeyondTxns.pd...

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#47

Post by newobj »

mercurial wrote: post #40
chiph wrote: post #35

Earlier quoted context omitted.

You could have a service bus, where you publish a "PersonDeleted" message that the other services would subscribe to. It decouples the Person service from all the other related entity services. You'd have to allow for propagation delay. Plus the possibility of a message storm if you delete something fairly fundamental.

> You could have a service bus, where you publish a "PersonDeleted" message that the other services would subscribe to. It decouples the Person service from all the other related entity services. You're still screwed if you complete a transaction on the deleted person's still-existing account, now that your system is no longer transactional...

Your objection is hypothetical/abstract and when you ground it in specific use cases there are plenty of patterns that emerge addressing how to deal with the inconsistent state. For example, just-in-time/read reconciliation, batch remediation, actually making some subset of actions transactional/consistent and suffering lower availability there, and so on and so forth.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#48

Post by newobj »

exacube wrote: post #15

This seems like it's just taking the idea of decoupling your service and talking to them via APIs a little further by saying.. decouple them to an even smaller granularity? This has always been the generally accepted way to scale out software services. Is there a novel idea being discussed here, or just that they've been doing this at Netflix?

When it comes down to it, n-tier, SOA, and microservices are all expressions of the same basic ideas.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#49

Post by threeseed »

davidkellis wrote: post #8
shanemhansen wrote: post #7

Earlier quoted context omitted.

Ideally you don't have to sync the data because one service owns that data. Other services request that data via api. In a RESTful world those api requests are cacheable.

But what about the situation where you have an entity service that owns the data for one piece of the domain, for example a People service, and then other services, like the Address service and the Billing service, reference a particular person. In that scenario, I can imagine the Address service and the Billing service would have a foreign key referencing a person in the People service. Then, what happens if the Per…

Look into Event Sourcing and CQRS.

It explains how to manage situations like this.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#50

Post by threeseed »

shanemhansen wrote: post #43
davidkellis wrote: post #8

Earlier quoted context omitted.

But what about the situation where you have an entity service that owns the data for one piece of the domain, for example a People service, and then other services, like the Address service and the Billing service, reference a particular person. In that scenario, I can imagine the Address service and the Billing service would have a foreign key referencing a person in the People service. Then, what happens if the Per…

So the problem you've identified is real. I used to have some bootleg footage of some private amazon tech talks where the speaker emphasized that in distributed systems it was generally a terrible idea to have transactions span entities. I think you basically have to learn to live in an eventually consistent world. In the case of people being deleted I would imagine that the user service exposes a pub/sub interface w…

You don't HAVE to live in an eventually consistent world. If you use something like ZeroMQ or use REST then you can "notify" other services of a "person deleted" event in a synchronous manner.
Post reply on HN