I was alive, however because I could not breath, I died.
Bob was fine himself, but someone shot him, so he is dead, (but remember bob was fine) --- What a joke
Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
241–250 of 410 posts
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#242Queue the armchair infrastructure engineers. The reality is that there’s a handful of people in the world that can operate systems at this sheer scale and complexity and I have mad respect for those in that camp.
Isn't this the equivalent of "complaining about your meal in a restaurant, I'd like to see you do better." The point of eating at a restaurant is that I can't/don't want to cook. Likewise, I use AWS because I want them to do the hard work and I'm willing to pay for it. How does that abrogate my right to complain if it goes badly (regardless of whether I could/couldn't do it myself)?
I think the distinction is you can say "I pay good money for you to do it properly and how dare you go down on me" but you become an "armchair infrastructure engineer" when you try and explain how you would have avoided the outage because you don't have the whole picture (especially based on a very carefully worded PR approved blog post).
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#243umm... But just one thing, S3 was not available at least for 20 minutes.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#244"Amazon Secure Token Service (STS) experienced elevated latencies" I was getting 503 "service unavailable" from STS during the outage most of the time I tried calling it. I guess by "elevated latency", they mean from anyone with retry logic that would keep trying after many consecutive attempts?
I suppose all outages are just elevated latency. Has anyone ever had an outage and said "fuck it, we're going out of business" and never came back up? That's the only true outage ;)
5xx errors are servers or proxies giving up on requests. Increased timeouts resulting in successful requests may have been considered "elevated latency" (but rarely this would be a proper way to solve similar issue).
They treat 5xx errors as non-errors but this is not the case with rest of the world. "Increased timeouts" is Amazon's untruthful term for "not working at all".
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#245I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…
> Third, you’re gonna go down when the cloud goes down.
Not necessarily. You just need to not be stuck with a single cloud provider. The likelihood of more than one availability zone going down on a single cloud provider is not that low in practice. Especially when the problem is a software bug.
The likelihood of AWS, Azure, and OVH going down at the same time is low. So if you need to stay online if AWS fail, don't put all your eggs in the AWS basket.
That means not using proprietary cloud solutions from a single cloud provider, it has a cost so it's not always worth it.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#246I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…
[deleted]
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#247I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…
Hey Wes!
I upvoted your comment before I noticed your handle.
+1 insightful, as usual
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#248I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…
> Third, you’re gonna go down when the cloud goes down. Not necessarily. You just need to not be stuck with a single cloud provider. The likelihood of more than one availability zone going down on a single cloud provider is not that low in practice. Especially when the problem is a software bug. The likelihood of AWS, Azure, and OVH going down at the same time is low. So if you need to stay online if AWS fail, don't…
> using proprietary cloud solutions from a single cloud provider, it has a cost so it's not always worth it.
but perhaps some software design choices could be made to alleviate these costs. For example, you could have a read-only replica on azure or whatever backup cloud provider, and design your software interfaces to allow the use of such read only replicas - at least you'd be degraded rather than unavailable. Ditto with web servers etc.
This has a cost, but it's lower than entirely replicating all of the proprietary features in a different cloud.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#249I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…
> Third, you’re gonna go down when the cloud goes down. Not necessarily. You just need to not be stuck with a single cloud provider. The likelihood of more than one availability zone going down on a single cloud provider is not that low in practice. Especially when the problem is a software bug. The likelihood of AWS, Azure, and OVH going down at the same time is low. So if you need to stay online if AWS fail, don't…
True multi-cloud redundancy is hard to test - because it’s everything from DNS on up and it’s hard to ask AWS to go offline so you can verify Azure picks up the slack.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#250I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…
Easy to say leave, the techinical lockin cloud service providers by design choose to have makes it impossible to leave . AWS (and others) make egress costs insanely expensive for any startup to consider leaving with their data, also there is constant push to either not support open protocols or extend /expand them in ways making it hard to migrate a code base easily. If the advise is to use only effectively use manag…
> AWS (and others) make egress costs insanely expensive for any startup to consider leaving with their data
I have seen this repeated many times, but don't understand it. Yes egress is expensive, but they are not THAT expensive compared to storage. S3 egress per GB is no more than 3x the price of storage, i.e. moving out just cost 3 month of storage cost (there's also API cost but that's not the one often mentioned).
Is egress pricing being a lock-in factor just a myth? Is there some other AWS cost I'm missing? Obviously there will be big architectural and engineering cost to move, but that's just part of life.