Earlier quoted context omitted.
It's the one with the chaos monkey built in.
More like Chaos Gorilla. https://en.wikipedia.org/wiki/Chaos_engineering#Chaos_Gorill...
Kind of like Donkey Kong, but he's rolling flaming barrels down the hot aisles.
AWS EC2 Having Issues
31–40 of 52 posts
Re: AWS EC2 Having Issues
#32GCP also experienced problems. A really nice one were if you deleted a service account you end up with restart loop: > We are experiencing an issue with Google Kubernetes Engine. Removing Service Account from GKE might lead to infinite cluster master restarts. Please refrain from removing GKE service accounts.
"Doctor, it hurts when I do this."
"Well, then don't do that"
Re: AWS EC2 Having Issues
#33Re: AWS EC2 Having Issues
#34interesting they use blue as the color that indicate an "issue". at a glance, looks like nothing to see here. i have a feeling that if the entire AWS infrastructure were to go offline somehow they would refer to that as 'increase error rates'
Downtime is kinda hard to grok for huge distributed services like this. If you’re down when anyone gets errors you’ll probably always be down — not really useful. If you’re down when everyone gets errors you’ll probably always be up — again not really useful. From a customer perspective an increase in errors means nothing if it doesn’t affect you or everything if it does so even a flat percentage is likely not all th…
Hence why they people use percentile measurements for alerting at places like AMZN.
This person worked on Alexa and wrote an article about it: https://medium.com/@djsmith42/how-to-metric-edafaf959fc7
Re: AWS EC2 Having Issues
#35interesting they use blue as the color that indicate an "issue". at a glance, looks like nothing to see here. i have a feeling that if the entire AWS infrastructure were to go offline somehow they would refer to that as 'increase error rates'
Interesting, I just noticed today Glassdoor changed the Negative Experience chart color from red to a more neutral bluish color.
Maybe it's a recent trend in the industry to distance themselves from negative traits? Amazon doesn't want you to think they're bad, Glassdor doesn't want their customers (corporations) to be seen in a bad light?
Re: AWS EC2 Having Issues
#36There actually are several good reasons to run in us-east-1.
Something I hear a good lot is the latency argument - if you're a startup based in Boston, and your roundtrip to us-west-2 is 80ms (this is actually my roundtrip to us-west 2 right now), it doesn't matter - your customers on the west coast will see 80ms if you go into us-east-1, too. That's true, but your first customers will probably be local, and you almost certainly don't have the resources to be doing true multi-region deployments right out of the gate. So my personal feeling is - deploy into us-east-1 to give your first customers a good experience - _fully understanding that you are taking on some extra risk_, and pay it down as tech debt in time.
Another plays off this, but in a very different way: a hybrid cloud deployment, where the public cloud is used as an on-demand extension of the datacenter. Something I once saw was that the 80ms round trip from a Boston datacenter to us-west-2 actually expanded to massive connection latencies: 80ms for a DNS lookup, and then another 500+ms (!) for the round-trips for TCP and TLS handshakes to take place, all before a SQL query or REST call actually started. That was a complete non-starter.
Re: AWS EC2 Having Issues
#37Re: AWS EC2 Having Issues
#38There actually are several good reasons to run in us-east-1. Something I hear a good lot is the latency argument - if you're a startup based in Boston, and your roundtrip to us-west-2 is 80ms (this is actually my roundtrip to us-west 2 right now), it doesn't matter - your customers on the west coast will see 80ms if you go into us-east-1, too. That's true, but your first customers will probably be local, and you almo…
That would all make sense, except that AWS has had a us-east-2 region in Ohio for a few years now.
Re: AWS EC2 Having Issues
#39friends don't let friends run on us-east-1
Re: AWS EC2 Having Issues
#40> Existing instances and networks continue to work normally.
This suggests that one can minimize certain failure modes by making the configuration as static as possible, i.e. creating a fixed number of long-running instances. But then, using immutable, ephemeral instances can make for a more resilient system when the EC2 control plane is working normally. There are always tradeoffs.