Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adopting Microservices at Netflix: Lessons for Architectural Design

nginx.com

Post reply on HN
Print view View original

11–20 of 76 posts

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#11

Post by davidkellis »

davidkellis wrote: post #8
shanemhansen wrote: post #7

Earlier quoted context omitted.

Ideally you don't have to sync the data because one service owns that data. Other services request that data via api. In a RESTful world those api requests are cacheable.

But what about the situation where you have an entity service that owns the data for one piece of the domain, for example a People service, and then other services, like the Address service and the Billing service, reference a particular person. In that scenario, I can imagine the Address service and the Billing service would have a foreign key referencing a person in the People service. Then, what happens if the Per…

cgh, I've hit the reply limit, but I wanted to ask how you'd implement the cascade deletes thing over services? Would the People service have to emit events describing that a person was deleted that the Address and Billing services would be expected to subscribe to in order to handle that the person was deleted?

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#12

Post by ryderm »

jfoutz wrote: post #6

If your app depends on lots of of them, it's only going to run as fast as the slowest dependency. 1% chance of poor performance isn't to bad, the joint distribution of 20 microservices each with a 1% chance, well that gets pretty ugly. In the normal case, everything is great, but the failure modes of each service become a much bigger deal. It's a great architecture, but fan out of dependencies is a real risk.

So you have short timeouts and retries that are load balanced to different nodes. But ideally your services are fast even in their 99 percentiles so this isn't an issue. This is much easier to achieve in a small service than a huge complex one.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#13

Post by cgh »

davidkellis wrote: post #11
davidkellis wrote: post #8

Earlier quoted context omitted.

But what about the situation where you have an entity service that owns the data for one piece of the domain, for example a People service, and then other services, like the Address service and the Billing service, reference a particular person. In that scenario, I can imagine the Address service and the Billing service would have a foreign key referencing a person in the People service. Then, what happens if the Per…

cgh, I've hit the reply limit, but I wanted to ask how you'd implement the cascade deletes thing over services? Would the People service have to emit events describing that a person was deleted that the Address and Billing services would be expected to subscribe to in order to handle that the person was deleted?

Sorry, I should have been more clear. I'm assuming a shared database. So cascading deletes would be defined in the table's schema. Let's pretend we're using Postgresql:

\d Person

[A bunch of table schema stuff]

Referenced by:

TABLE "Billing" CONSTRAINT "billing_id_fk" FOREIGN KEY (id) REFERENCES Person(id) ON DELETE CASCADE

(I typed that off the type of my head so it might not be quite correct.)

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#14

Post by ryderm »

davidkellis wrote: post #11
davidkellis wrote: post #8

Earlier quoted context omitted.

But what about the situation where you have an entity service that owns the data for one piece of the domain, for example a People service, and then other services, like the Address service and the Billing service, reference a particular person. In that scenario, I can imagine the Address service and the Billing service would have a foreign key referencing a person in the People service. Then, what happens if the Per…

cgh, I've hit the reply limit, but I wanted to ask how you'd implement the cascade deletes thing over services? Would the People service have to emit events describing that a person was deleted that the Address and Billing services would be expected to subscribe to in order to handle that the person was deleted?

Yeah, events. Which are message queues for most things, api calls if it really needs to be done synchronously

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#15

Post by exacube »

This seems like it's just taking the idea of decoupling your service and talking to them via APIs a little further by saying.. decouple them to an even smaller granularity?

This has always been the generally accepted way to scale out software services. Is there a novel idea being discussed here, or just that they've been doing this at Netflix?

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#16

Post by nawitus »

A few questions:

a) How do you prevent technical debt? It seems to be more difficult due to APIs which shouldn't have breaking changes. In theory you could always version up the APIs and serve both versions or just add a new API for a breaking change, but these solutions seems awkward.

b) How do you start developing multiple microservices at the same time? I would expect APIs to change a lot in the beginning, which would mean that updating one microservice would break another. Perhaps that is acceptable before the first "stable release" of a microservice.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#17

Post by dsp1234 »

cgh wrote: post #13
davidkellis wrote: post #11

Earlier quoted context omitted.

cgh, I've hit the reply limit, but I wanted to ask how you'd implement the cascade deletes thing over services? Would the People service have to emit events describing that a person was deleted that the Address and Billing services would be expected to subscribe to in order to handle that the person was deleted?

Sorry, I should have been more clear. I'm assuming a shared database. So cascading deletes would be defined in the table's schema. Let's pretend we're using Postgresql: \d Person [A bunch of table schema stuff] Referenced by: TABLE "Billing" CONSTRAINT "billing_id_fk" FOREIGN KEY (id) REFERENCES Person(id) ON DELETE CASCADE (I typed that off the type of my head so it might not be quite correct.)

The original position being argued though was "... where all services talk to the same database... You need to split the database up and denormalize it.".

So the basic premise is that there is no shared database, and thus having the database enforce cascading deletes is not an option.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#18

Post by bigdubs »

nawitus wrote: post #16

A few questions: a) How do you prevent technical debt? It seems to be more difficult due to APIs which shouldn't have breaking changes. In theory you could always version up the APIs and serve both versions or just add a new API for a breaking change, but these solutions seems awkward. b) How do you start developing multiple microservices at the same time? I would expect APIs to change a lot in the beginning, which w…

Honestly, I've been following along with these trends a lot and it seems like the answers for 1&2 come down to team size. If you have a bigger team, with more development inertia microservices can seem amazing and the tradeoffs are worth it (repeated work vs. development tempo).

To a small team that doesn't have the inertial issues to generate the benefits of microservices, it seems like they are nice in theory but have too much overhead to supplant monolithic approaches.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#19

Post by jfoutz »

ryderm wrote: post #12
jfoutz wrote: post #6

If your app depends on lots of of them, it's only going to run as fast as the slowest dependency. 1% chance of poor performance isn't to bad, the joint distribution of 20 microservices each with a 1% chance, well that gets pretty ugly. In the normal case, everything is great, but the failure modes of each service become a much bigger deal. It's a great architecture, but fan out of dependencies is a real risk.

So you have short timeouts and retries that are load balanced to different nodes. But ideally your services are fast even in their 99 percentiles so this isn't an issue. This is much easier to achieve in a small service than a huge complex one.

Um. maybe. 5 machines behind a load balancer. Normal case, load is even, 100 requests to each server. One server starts running into trouble, exceeding timeouts. your load is now ~125 per server, because each client retries frequently. Is 125 enough to push over a "slow" threshold on the others? This will further magnify the load.

The load balancer will spin up more machines, so now you have 10 machines leaning on whatever the back end is.

Yes, your approach is great - but you really have to understand the failure modes - if you're living on the edge, you could have a pretty un fun cascading error.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#20

Post by jacques_chester »

nawitus wrote: post #16

A few questions: a) How do you prevent technical debt? It seems to be more difficult due to APIs which shouldn't have breaking changes. In theory you could always version up the APIs and serve both versions or just add a new API for a breaking change, but these solutions seems awkward. b) How do you start developing multiple microservices at the same time? I would expect APIs to change a lot in the beginning, which w…

> How do you start developing multiple microservices at the same time?

Same as any other project: Develop from the outside in.

In practice, trying to develop in the "optimal order' leads to speculative development that will be wasted.

Post reply on HN