Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing Chaos Engineering

techblog.netflix.com

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Introducing Chaos Engineering

#11

Post by opendais »

Kronopath wrote: post #10
herge wrote: post #4

Man, I really have the impression that my video-on-demand service has a better understanding of ensuring availability and risk-management than either my bank or any government service. Could you imagine say a utility company creating a title called 'Chaos Engineer'.

Can you imagine how difficult that would be to justify in a bank or government service? It would be very hard to convince someone in a position of power at an institution like that that deliberately introducing random failures and problems into your production services would be a good idea. Even if it is a good idea, it would be hard to make that argument. If Netflix's system fails, the worst thing that could happen…

You run a staging environment to mirror production [with fewer nodes per PoP] and throw things at it.

The only real difference is the load on staging would be generated artificially [e.g. ACH, credit card processing] using faked staging-only accounts. You even have a fake banking website for security audits that is a clone of production with the fake accounts.

No risk to production and probably 80% of the benefits. Of course its probably 125% of the effort.

Re: Introducing Chaos Engineering

#13

Post by [deleted] »

herge wrote: post #4

Man, I really have the impression that my video-on-demand service has a better understanding of ensuring availability and risk-management than either my bank or any government service. Could you imagine say a utility company creating a title called 'Chaos Engineer'.

[deleted]

Re: Introducing Chaos Engineering

#14

Post by angersock »

herge wrote: post #4

Man, I really have the impression that my video-on-demand service has a better understanding of ensuring availability and risk-management than either my bank or any government service. Could you imagine say a utility company creating a title called 'Chaos Engineer'.

4chan has better availability and higher traffic than healthcare.gov. Think about that.

Re: Introducing Chaos Engineering

#15

Post by Spooky23 »

Kronopath wrote: post #10
herge wrote: post #4

Man, I really have the impression that my video-on-demand service has a better understanding of ensuring availability and risk-management than either my bank or any government service. Could you imagine say a utility company creating a title called 'Chaos Engineer'.

Can you imagine how difficult that would be to justify in a bank or government service? It would be very hard to convince someone in a position of power at an institution like that that deliberately introducing random failures and problems into your production services would be a good idea. Even if it is a good idea, it would be hard to make that argument. If Netflix's system fails, the worst thing that could happen…

I work for a large institution. The thought process doesn't work that way -- the suggestion of introducing chaos would be a no-go not because of business risk, but the risk of disrupting the layers of bullshit that people build between them and whatever the actual business is.

Netflix's cost of failure is very high -- customers get real-time discovery of your failures, and the service costs less than $10/mo, so they aren't deeply invested in it and will drop you like a hot potato.

Re: Introducing Chaos Engineering

#16

Post by kooshball »

frozenport wrote: post #9
kperry wrote: post #6

Netflix puts out some great articles about architecture in the cloud. Auto-scaling, chaos monkey, and how they handle 'steal-time.' Does anyone know of any other company that publishes so much about cloud architecture? This is great stuff!

Most companies their size would use their own servers instead of the cloud.

Using AWS does not magically give you a HA infrastructure when you have a complicated service oriented architecture like Netflix. All the stuff mentioned here are still relevant even if they're running their own DC.

Re: Introducing Chaos Engineering

#17

Post by mkoller »

kperry wrote: post #6

Netflix puts out some great articles about architecture in the cloud. Auto-scaling, chaos monkey, and how they handle 'steal-time.' Does anyone know of any other company that publishes so much about cloud architecture? This is great stuff!

Check out the CloudFlare blog http://blog.cloudflare.com/ Here is a good read ( http://blog.cloudflare.com/technical-details-behind-a-400gbp... )

Re: Introducing Chaos Engineering

#18

Post by diminoten »

opendais wrote: post #11
Kronopath wrote: post #10

Earlier quoted context omitted.

Can you imagine how difficult that would be to justify in a bank or government service? It would be very hard to convince someone in a position of power at an institution like that that deliberately introducing random failures and problems into your production services would be a good idea. Even if it is a good idea, it would be hard to make that argument. If Netflix's system fails, the worst thing that could happen…

You run a staging environment to mirror production [with fewer nodes per PoP] and throw things at it. The only real difference is the load on staging would be generated artificially [e.g. ACH, credit card processing] using faked staging-only accounts. You even have a fake banking website for security audits that is a clone of production with the fake accounts. No risk to production and probably 80% of the benefits. O…

Show me a staging environment that's as robust as a production environment...

Re: Introducing Chaos Engineering

#20

Post by [deleted] »

hortonew wrote: post #8
herge wrote: post #4

Man, I really have the impression that my video-on-demand service has a better understanding of ensuring availability and risk-management than either my bank or any government service. Could you imagine say a utility company creating a title called 'Chaos Engineer'.

It's so true. The entertainment/consumer shopping industries seems to have surpassed banks/governments in availability. It's also pretty ridiculous knowing that some of these companies are tasked with keeping our data/money safe, and yet they still don't allow extremely complex passwords.

[deleted]
Post reply on HN