Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adopting Microservices at Netflix: Lessons for Architectural Design

nginx.com

Post reply on HN
Print view View original

21–30 of 76 posts

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#21

Post by cgh »

dsp1234 wrote: post #17
cgh wrote: post #13

Earlier quoted context omitted.

Sorry, I should have been more clear. I'm assuming a shared database. So cascading deletes would be defined in the table's schema. Let's pretend we're using Postgresql: \d Person [A bunch of table schema stuff] Referenced by: TABLE "Billing" CONSTRAINT "billing_id_fk" FOREIGN KEY (id) REFERENCES Person(id) ON DELETE CASCADE (I typed that off the type of my head so it might not be quite correct.)

The original position being argued though was "... where all services talk to the same database... You need to split the database up and denormalize it." . So the basic premise is that there is no shared database, and thus having the database enforce cascading deletes is not an option.

Right, thanks for the clarification. Sorry for the slight derailment.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#22

Post by samspot »

jfoutz wrote: post #6

If your app depends on lots of of them, it's only going to run as fast as the slowest dependency. 1% chance of poor performance isn't to bad, the joint distribution of 20 microservices each with a 1% chance, well that gets pretty ugly. In the normal case, everything is great, but the failure modes of each service become a much bigger deal. It's a great architecture, but fan out of dependencies is a real risk.

I don't find that performance runs at % chance scale. When things perform poorly, they are often very consistent. With this architecture you can rapidly iterate & scale your poor performing core services, while the non-core services can run slowly w/o causing a big deal.

The usual alternative is redeploying and scaling the entire app to iterate on performance, which is a much slower process. If performance is a concern, microservices should be a big win.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#23

Post by sarnowski »

jfoutz wrote: post #19
ryderm wrote: post #12

Earlier quoted context omitted.

So you have short timeouts and retries that are load balanced to different nodes. But ideally your services are fast even in their 99 percentiles so this isn't an issue. This is much easier to achieve in a small service than a huge complex one.

Um. maybe. 5 machines behind a load balancer. Normal case, load is even, 100 requests to each server. One server starts running into trouble, exceeding timeouts. your load is now ~125 per server, because each client retries frequently. Is 125 enough to push over a "slow" threshold on the others? This will further magnify the load. The load balancer will spin up more machines, so now you have 10 machines leaning on wh…

Thats what circuit breakers are for to not have cascading errors.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#24

Post by HeyLaughingBoy »

jacques_chester wrote: post #20
nawitus wrote: post #16

A few questions: a) How do you prevent technical debt? It seems to be more difficult due to APIs which shouldn't have breaking changes. In theory you could always version up the APIs and serve both versions or just add a new API for a breaking change, but these solutions seems awkward. b) How do you start developing multiple microservices at the same time? I would expect APIs to change a lot in the beginning, which w…

> How do you start developing multiple microservices at the same time? Same as any other project: Develop from the outside in. In practice, trying to develop in the "optimal order' leads to speculative development that will be wasted.

I really wish the team I'm on could understand this!

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#25

Post by Jormundir »

jfoutz wrote: post #6

If your app depends on lots of of them, it's only going to run as fast as the slowest dependency. 1% chance of poor performance isn't to bad, the joint distribution of 20 microservices each with a 1% chance, well that gets pretty ugly. In the normal case, everything is great, but the failure modes of each service become a much bigger deal. It's a great architecture, but fan out of dependencies is a real risk.

How would you do it?

Bringing those dependencies in under one roof doesn't eliminate their risk. Though it does increase the chance that an error in one of these small dependencies brings down the whole system.

With microservices if one of the services runs into trouble, you can ignore it and still serve the other 90-99% of your site without it. You can also deploy updates to services without having to deploy your entire site.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#26

Post by mercurial »

jacques_chester wrote: post #20
nawitus wrote: post #16

A few questions: a) How do you prevent technical debt? It seems to be more difficult due to APIs which shouldn't have breaking changes. In theory you could always version up the APIs and serve both versions or just add a new API for a breaking change, but these solutions seems awkward. b) How do you start developing multiple microservices at the same time? I would expect APIs to change a lot in the beginning, which w…

> How do you start developing multiple microservices at the same time? Same as any other project: Develop from the outside in. In practice, trying to develop in the "optimal order' leads to speculative development that will be wasted.

Nothing prevents you from having multiple microservices sharing the same codebase.

That said, the "it's just like the web" model doesn't sound fantastic to me. It sounds like your app now depends on contracts which are only enforced by good practices, not by something strongly typed you can check at compile time, unless you use something like protocol buffers to generate the boilerplate.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#27

Post by tjdetwiler »

samspot wrote: post #22
jfoutz wrote: post #6

If your app depends on lots of of them, it's only going to run as fast as the slowest dependency. 1% chance of poor performance isn't to bad, the joint distribution of 20 microservices each with a 1% chance, well that gets pretty ugly. In the normal case, everything is great, but the failure modes of each service become a much bigger deal. It's a great architecture, but fan out of dependencies is a real risk.

I don't find that performance runs at % chance scale. When things perform poorly, they are often very consistent. With this architecture you can rapidly iterate & scale your poor performing core services, while the non-core services can run slowly w/o causing a big deal. The usual alternative is redeploying and scaling the entire app to iterate on performance, which is a much slower process. If performance is a conce…

All APIs have performance SLAs that are not consistent. You may have 50% of requests finish in 200ms or less, 90% finish in 300ms or less, and 99% finish in 600ms or less. You can do work to narrow the performance variation but performance is a % chance.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#28

Post by danudey »

davidkellis wrote: post #8
shanemhansen wrote: post #7

Earlier quoted context omitted.

Ideally you don't have to sync the data because one service owns that data. Other services request that data via api. In a RESTful world those api requests are cacheable.

But what about the situation where you have an entity service that owns the data for one piece of the domain, for example a People service, and then other services, like the Address service and the Billing service, reference a particular person. In that scenario, I can imagine the Address service and the Billing service would have a foreign key referencing a person in the People service. Then, what happens if the Per…

The People service can also store Addresses. Call it the Identity service. Include People, Businesses, Relationships, and Addresses.

The Billing service can then reference People or Businesses (and if Businesses, then sub-People), bill to an Address, etc.

No one's saying every object should be a service; you need to find the correct lines to divide across.

In our system (which has been service-oriented for five years), we don't do deletes. We do 'inactive' (UPDATE table SET ACTIVE=0…), but never deletes.

Especially in a case of your billing example, you never want to delete a person or address, because that's historical data you need to retain, but we just keep everything. If it goes in the database, it's because we want to keep it forever.

Re: Adopting Microservices at Netflix: Lessons for Architectural Design

#29

Post by danudey »

jfoutz wrote: post #6

If your app depends on lots of of them, it's only going to run as fast as the slowest dependency. 1% chance of poor performance isn't to bad, the joint distribution of 20 microservices each with a 1% chance, well that gets pretty ugly. In the normal case, everything is great, but the failure modes of each service become a much bigger deal. It's a great architecture, but fan out of dependencies is a real risk.

This is solved because you can scale each service differently. Too many DB queries from your messaging service? Upgrade the DB, add another read slave. Too much CPU load on your image processing service? Add more image processing nodes.

Breaking your system out into multiple dependencies means you can not only scale your infrastructure, but you can scale individual parts of your infrastructure based on demand, bottlenecks, usage, etc.

Netflix has talked about in the past how, because their systems are broken apart, they don't have to deal with these issues. Rating service having problems? Don't show user ratings. Search service offline for updates? Disable search. If Netflix was one giant (Rails? Django? Node?) app, it would be very difficult to cut out poorly-performing parts temporarily.

Post reply on HN