Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adopting Microservices at Netflix: Lessons for Architectural Design

nginx.com

Post reply on HN
Print view View original

61–70 of 76 posts

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#61

Post by alxndr »

polynomeal wrote: post #57

My team recently added a microservice to support our fairly monolythic backend service. The big challenge we found was that it takes a lot of effort to make a new (micro)service. We need to create a system of alarms (instead of relying on existing catch all defaults). We need its own test environment, we need to find ways to send traffic to pre-prod. We needed to figure out how to bootstrap the new service into the c…

I've had the same experience lately. Coordinating a number of staging environments for an evolving SOA backend has been challenging for both developers and ops. In addition to the services, each one can have many other things to worry about: monitoring, error collection, which version is deployed to which environment, replicating the complexity locally to work on it...

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#62

Post by alxndr »

jacques_chester wrote: post #33
danudey wrote: post #31

Earlier quoted context omitted.

This is basically it. Designing a game? Build the Game service, with all your game logic. Need users and authentication now? Start writing an Identity service, and so on. The only difference is that instead of starting to write some Identity class and use it in your Game service, you write some Identity class and expose it via a REST API, and then provide an interface library that interfaces with that REST API. Call…

The problem is that microservices are not objects. They leak reality into your problem domain in a way that simply cannot be made to go away. If regular object oriented programming languages had method calls that randomly failed, were delayed, sent multiple copies of a response, changed how they behaved without warning, sent half-formed responses ... then yes it would be the same. Distributed systems are hard, becaus…

One of the things I've heard about Erlang is that its processes can be distributed very easily. The paper you cite was written four years before Erlang was open-sourced; I wonder if Erlang/OTP would hold up to their analysis.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#63

Post by nl »

davidw wrote: post #2

My comment, slightly edited, from the previous posting of this, at https://news.ycombinator.com/item?id=9106813 > One kind of coupling that people tend to overlook as they transition to a microservices architecture is database coupling, where all services talk to the same database and updating a service means changing the schema. You need to split the database up and denormalize it. That sounds like a decision you wo…

Database-as-an-integration-layer is a well known anti-pattern.

It's tempting because it's easy, and at first glance it seems to solve lots of problems (consistency, communication etc).

It's a big mistake.

If multiple service are reading and writing from the same DB it rapidly becomes impossible to change things.

Things like input validation changes suddenly have to be implemented in multiple places (which is hard), and done simultaneously (which often becomes hard enough to stop work being done).

In a microservice based approach, the data flows though shared services, and so changes can be done in fewer places.

(Note that read-only, reporting-style databases are separate. I think there is a good case for these being shared)

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#64

Post by dragonwriter »

nl wrote: post #63
davidw wrote: post #2

My comment, slightly edited, from the previous posting of this, at https://news.ycombinator.com/item?id=9106813 > One kind of coupling that people tend to overlook as they transition to a microservices architecture is database coupling, where all services talk to the same database and updating a service means changing the schema. You need to split the database up and denormalize it. That sounds like a decision you wo…

Database-as-an-integration-layer is a well known anti-pattern. It's tempting because it's easy, and at first glance it seems to solve lots of problems (consistency, communication etc). It's a big mistake. If multiple service are reading and writing from the same DB it rapidly becomes impossible to change things. Things like input validation changes suddenly have to be implemented in multiple places (which is hard), a…

Multiple services reading from the same DB can -- and should -- be using different accounts that have access to different objects that abstract the underlying data model from the various clients and keep them loosely coupled. This is almost as old of a recommendation as relational DBs themselves.

You can create a tightly coupled design with a shared DB, but there is nothing inherent with shared DB integration requires that.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#65

Post by dragonwriter »

davidkellis wrote: post #8
shanemhansen wrote: post #7

Earlier quoted context omitted.

Ideally you don't have to sync the data because one service owns that data. Other services request that data via api. In a RESTful world those api requests are cacheable.

But what about the situation where you have an entity service that owns the data for one piece of the domain, for example a People service, and then other services, like the Address service and the Billing service, reference a particular person. In that scenario, I can imagine the Address service and the Billing service would have a foreign key referencing a person in the People service. Then, what happens if the Per…

Probably, a Person shouldn't be deleted if it might be referenced elsewhere. If it's no longer a valid customer, it should be updated to reflect that.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#66

Post by mercurial »

jacques_chester wrote: post #45
mercurial wrote: post #44

Earlier quoted context omitted.

> As soon as you distribute your system, you have dynamic typing, whether you like it or not. You have serialization/deserialization issues. You can still type your messages. > At runtime you are inspecting incoming messages and then routing them to code. It doesn't matter what language the code is written in, it will need to route and validate the messages at runtime. Of course. > The type system cannot provide comp…

> You can still type your messages. You can hope that they respect the type. For a robust distributed system, you will have to check everything at runtime. > If you make the assumption that you deploy up-to-date binaries, then knowing at compile time that your producer and consumer use the same data structure for the messages they exchange would give me much better confidence than "it looks like the API conforms to w…

If you cannot ensure that your producer receives messages following a certain schema, even though you enforce it statically in your codebase, you also cannot ensure that your running code passes your tests.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#67

Post by mercurial »

newobj wrote: post #47
mercurial wrote: post #40

Earlier quoted context omitted.

> You could have a service bus, where you publish a "PersonDeleted" message that the other services would subscribe to. It decouples the Person service from all the other related entity services. You're still screwed if you complete a transaction on the deleted person's still-existing account, now that your system is no longer transactional...

Your objection is hypothetical/abstract and when you ground it in specific use cases there are plenty of patterns that emerge addressing how to deal with the inconsistent state. For example, just-in-time/read reconciliation, batch remediation, actually making some subset of actions transactional/consistent and suffering lower availability there, and so on and so forth.

I'm not saying there are no solutions, but saying "just add an event bus" is unlikely to be sufficient. Whatever you do, you're going to pay additional costs in terms of complexity.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#68

Post by reubenbond »

shanemhansen wrote: post #54
pmahoney wrote: post #52

Earlier quoted context omitted.

> Rating service having problems? Don't show user ratings. Search service offline for updates? Disable search As an example of a (probably?) bad way to organize services, I worked on a project that had factored a role-based access control system into its own service. Every single web request hit this service, which made it a single point of failure, performance critical, impossible to temporarily disable, etc.

One alternative to centralized role servers is to use client certificates. I've used x509 certs for this purpose. They are pretty hairy, but so is rolling your own authentication/authorization/token system.

Another alternative is JSON Web Tokens. Many of the benefits of Client Certificates while avoiding many of the hardships.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#69

Post by sanmon3186 »

davidkellis wrote: post #8
shanemhansen wrote: post #7

Earlier quoted context omitted.

Ideally you don't have to sync the data because one service owns that data. Other services request that data via api. In a RESTful world those api requests are cacheable.

But what about the situation where you have an entity service that owns the data for one piece of the domain, for example a People service, and then other services, like the Address service and the Billing service, reference a particular person. In that scenario, I can imagine the Address service and the Billing service would have a foreign key referencing a person in the People service. Then, what happens if the Per…

In my experience, you only expose APIs that are either standalone and transaction-ally independent (change address, delete address etc. in your example) or composite services (say people service) that should manage this distributed transaction. How transactions are managed under the hood vary based on implementation.

One may argue that in this case, "people" service doesn't go by description of micro-service as given in the article. But we need to understand that services get called in some context and there has to be someone there to do the plumbing. That someone can either be a db query, some code in the service, or app/application calling the services. And "generally" you would prefer service code over other two and hence a composite service.

IMO it may also be okay to have People, address, billing under one schema if service granularity and context allows so.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#70

Post by jacques_chester »

mercurial wrote: post #66
jacques_chester wrote: post #45

Earlier quoted context omitted.

> You can still type your messages. You can hope that they respect the type. For a robust distributed system, you will have to check everything at runtime. > If you make the assumption that you deploy up-to-date binaries, then knowing at compile time that your producer and consumer use the same data structure for the messages they exchange would give me much better confidence than "it looks like the API conforms to w…

If you cannot ensure that your producer receives messages following a certain schema, even though you enforce it statically in your codebase, you also cannot ensure that your running code passes your tests.

Which is why I start from integration testing of the whole system, with frenemy tests for any foreign services that I must rely on.

You're right that tests don't make Byzantine failures go away. But neither do static types. My point that distribution turns all systems into analogies for dynamic language programming remains, and so the emphasis on tool support changes along with it.

Post reply on HN