21–30 of 31 posts
I wonder if Netflix has ever come out with some kind of, "So you want to get into chaos engineering, eh?" kind of article that explains the basics and some pitfalls/things to look out for.
Man, I really have the impression that my video-on-demand service has a better understanding of ensuring availability and risk-management than either my bank or any government service. Could you imagine say a utility company creating a title called 'Chaos Engineer'.
How does one go about becoming a chaos engineer? I imagine up to this point it is a field one falls into accidentally and gains experience over time. I can imagine it becoming a topic taught at a college level in the near future.
You just need an appreciation that any down-time in a system is a symptom of larger problems, and the will to identify (and reproduce as chaos!) those problems.
Earlier quoted context omitted.
You run a staging environment to mirror production [with fewer nodes per PoP] and throw things at it. The only real difference is the load on staging would be generated artificially [e.g. ACH, credit card processing] using faked staging-only accounts. You even have a fake banking website for security audits that is a clone of production with the fake accounts. No risk to production and probably 80% of the benefits. O…
Show me a staging environment that's as robust as a production environment...
Yeah, ok that had nothing to do with the OP, sorry, I just had to vent.
I wonder if Netflix has ever come out with some kind of, "So you want to get into chaos engineering, eh?" kind of article that explains the basics and some pitfalls/things to look out for.
I gave a talk at pagerduty about how this has been done for the last few years: https://blog.pagerduty.com/2014/03/injecting-failure-at-netf...
Earlier quoted context omitted.
Show me a staging environment that's as robust as a production environment...
My last company I worked at, they had like 30% of the staging environment running production stuff. They also had only one deploy target, which is drum roll production, which meant all of staging AND some of Dev/QA was live and could interact with the production environment. Also fun to see the production load at 300% of expected, burning, only to find out some event, data roll up, query, whatever was being run in du…
The kinds of things I work on now are entire "environments" we sell, so "production" testing is more along the lines of "go plug the box in and do stuff to it".
Earlier quoted context omitted.
You run a staging environment to mirror production [with fewer nodes per PoP] and throw things at it. The only real difference is the load on staging would be generated artificially [e.g. ACH, credit card processing] using faked staging-only accounts. You even have a fake banking website for security audits that is a clone of production with the fake accounts. No risk to production and probably 80% of the benefits. O…
Show me a staging environment that's as robust as a production environment...
2) The projects I work at on $DAY_JOB that I control all have 3 hardware nodes for HA in both staging & production. Staging is under artificial load that plays back the same series of events on dummy data. [e.g. Job A runs -> Completes -> Puts a mock job in Staging's Job Queue -> runs on Staging with mocked data -> I randomly restart a node to see what happens]
Buying 3 1Us is $750 for a staging environment. If you have 1 DC availability, you can just put them on your Local LAN (Free).
http://www.ebay.com/itm/Sun-Oracle-x4140-1U-Server-2x-Optero...
Now, you suddenly have a rough approximation of a 3 Node / 1 DC environment.
But what if you have multiple DCs, 3 Nodes per DC, Hot/Cold Loadbalancers and have DNS Failover Between Them?!
Say, you run a LBx2+Nodex3/LBx2+Nodex3/LBx2+Nodex3 across 3 DCs.
Assuming you automate the build, you could just pay by hour. But if not...
Linode's Failover IPs let you do the Hot/Cold LB thing [$60 for 3 DCs worth]. 3 Nodes per DC [$40 Nodes for 4096 MB is probably more than enough to test most issues]. $360
So for $420/month, you could replicate a pretty big real world setup.
What if you have 25 servers like stackoverflow and a hot/cold DC setup that is all Linux? http://highscalability.com/blog/2014/7/21/stackoverflow-upda...
Linode's Failover IPs let you do the Hot/Cold LB thing [$40 for 2 DCs worth]. 13 Nodes per DC [$40 Nodes for 4096 MB is probably more than enough to test most issues]. $520
$560/month you can basically duplicate the 2 coast hot/cold setup with 26 servers.
But its expensive!
...unless this is a one-man operation, not really. A fully loaded junior engineer is a minimum of $120k. A robust staging environment is very useful and isn't going to cost more than 1 engineer month to setup.
Yes, you might not use the same hardware. However, it is close enough and honestly...the same hardware 1:1 with real servers [based on the ebay servers] and a rack at Fremont w/ Hurricane Electric is $600/month. 13 servers is $3,250 in hardware. Get someone else on the East Coast in like Buffalo, NY and you can get as close as your budget affords. I doubt you'd hit 1 Junior Engineer Year worth of $$.
Not really Free. Power and A/C are things you'll pay for but its not a notable cost [less than $100/month].
Man, I really have the impression that my video-on-demand service has a better understanding of ensuring availability and risk-management than either my bank or any government service. Could you imagine say a utility company creating a title called 'Chaos Engineer'.
It's so true. The entertainment/consumer shopping industries seems to have surpassed banks/governments in availability. It's also pretty ridiculous knowing that some of these companies are tasked with keeping our data/money safe, and yet they still don't allow extremely complex passwords.
In the end, one system needs to be responsible for a given account... best case you could have is only some accounts/users are affected. It's a different kind of problem.
Facebook, for example uses Cassandra with a very wide distribution of nodes, with an immediate ack on receive... you can't do that with a bank... transactional payment systems won't tolerate it. You could do a few things to alleviate this issue... read-only replicas, etc... even then it's a matter of failing soft (disabling only those systems/services that are unavailable instead of the whole thing).
Earlier quoted context omitted.
Can you imagine how difficult that would be to justify in a bank or government service? It would be very hard to convince someone in a position of power at an institution like that that deliberately introducing random failures and problems into your production services would be a good idea. Even if it is a good idea, it would be hard to make that argument. If Netflix's system fails, the worst thing that could happen…
I work for a large institution. The thought process doesn't work that way -- the suggestion of introducing chaos would be a no-go not because of business risk, but the risk of disrupting the layers of bullshit that people build between them and whatever the actual business is. Netflix's cost of failure is very high -- customers get real-time discovery of your failures, and the service costs less than $10/mo, so they…