House of cards
Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
261–270 of 410 posts
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#262I’ve been running platform teams on aws now for 10 years, and working in aws for 13. For anyone looking for guidance on how to avoid this, here’s the advice I give startups I advise. First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region. Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up…
Or rent bare metal servers like old times and be responsible for your own s*t
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#263> This resulted in a large surge of connection activity that overwhelmed the networking devices between the internal network and the main AWS network, resulting in delays for communication between these networks. These delays increased latency and errors for services communicating between these networks, resulting in even more connection attempts and retries. This led to persistent congestion and performance issues o…
But what's a good alternative then? What if the internet connection has recovered? And you were at the, for example, 4 minute retry loop. Would you just make your users stare at a spinning loader for 8 minutes?
I first learned about exponential backoff from TCP and TCP has a lot of other smart ways to manage congestion control. You don't need to implement all the ideas into client logic but you can also do a lot better than just basic exponential backoff.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#264> This resulted in a large surge of connection activity that overwhelmed the networking devices between the internal network and the main AWS network, resulting in delays for communication between these networks. These delays increased latency and errors for services communicating between these networks, resulting in even more connection attempts and retries. This led to persistent congestion and performance issues o…
But what's a good alternative then? What if the internet connection has recovered? And you were at the, for example, 4 minute retry loop. Would you just make your users stare at a spinning loader for 8 minutes?
Sure, why not?
Or tell them directly that "We have screwed up. The service is currently overloaded. Thank you for your patience. If you still haven't given up on us, try again a less busy time of day. We are very sorry."
There are several options, and finding the best one depends a bit on estimating the behaviour of your specific target audience.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#265Earlier quoted context omitted.
Easy to say leave, the techinical lockin cloud service providers by design choose to have makes it impossible to leave . AWS (and others) make egress costs insanely expensive for any startup to consider leaving with their data, also there is constant push to either not support open protocols or extend /expand them in ways making it hard to migrate a code base easily. If the advise is to use only effectively use manag…
> AWS (and others) make egress costs insanely expensive for any startup to consider leaving with their data I have seen this repeated many times, but don't understand it. Yes egress is expensive, but they are not THAT expensive compared to storage. S3 egress per GB is no more than 3x the price of storage, i.e. moving out just cost 3 month of storage cost (there's also API cost but that's not the one often mentioned).…
3months is only if you use standard S3, However intelligent tiering , infrequent access , reduced redundancy or glacier instant can be substantially cheaper, without impacting retrieval time [1]
At scale when costs matter, you would have lifecycle policy tuned to your needs taking advantage of these classes. Any typical production workload is hardly paying only S3 base price for all/most of its storage needs, they will have mix of all these too.
[1] if there is substantial data in glacier regular, the costing completely blows through the roof, retrieval +egress makes it infeasible unless you activily hate AWS enough to spend that kind of money
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#266> This resulted in a large surge of connection activity that overwhelmed the networking devices between the internal network and the main AWS network, resulting in delays for communication between these networks. These delays increased latency and errors for services communicating between these networks, resulting in even more connection attempts and retries. This led to persistent congestion and performance issues o…
> System dynamics are hard. And have to be actually tested. Most of them are designs based on nothing but uninformed intuition. There is an art to back pressure and keeping pipelines optimally utilized. Queueing doesn’t work like you think until you really know.
Why is this hard, and can’t just be written down somewhere as part of the engineering discipline? This aspect of systems in 2021 really shouldn’t be an “art.”
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#267Earlier quoted context omitted.
> Third, you’re gonna go down when the cloud goes down. Not necessarily. You just need to not be stuck with a single cloud provider. The likelihood of more than one availability zone going down on a single cloud provider is not that low in practice. Especially when the problem is a software bug. The likelihood of AWS, Azure, and OVH going down at the same time is low. So if you need to stay online if AWS fail, don't…
True multi-cloud redundancy is hard to test - because it’s everything from DNS on up and it’s hard to ask AWS to go offline so you can verify Azure picks up the slack.
Sure you can. Firewall AWS off from whatever machine does the health checks in the redundancy implementation.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#268Earlier quoted context omitted.
Easy to say leave, the techinical lockin cloud service providers by design choose to have makes it impossible to leave . AWS (and others) make egress costs insanely expensive for any startup to consider leaving with their data, also there is constant push to either not support open protocols or extend /expand them in ways making it hard to migrate a code base easily. If the advise is to use only effectively use manag…
Lesson to build your services with Docker and Terraform. In this setup you can spin up a working clone of a decently sized stack in a different cloud provider in under an hour. Don't lock yourself in.
If the setup is that portable you probably don't need the AWS at all in the first place.
If your use only services built and managed by your docker images why use the cloud in the first place ? It would be cheaper to host on a smaller vendor , the reliability is not substantially better with big cloud than tier two vendors, that difference between say OVH and AWS is not that valuable to most applications to be worth the premium.
In IMO, if you don't leverage cloud native services offered by GCP or AWS then cloud is not adding much value to your stack.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#269Earlier quoted context omitted.
> AWS in general is trying hard to do the right thing for customers You are responding to a comment that suggests they're misrepresenting the truth (which wouldn't be the first time even in last few days) in communication to their customers. As always, they are doing the right thing for themselves only. EDIT: I think that you should mention being an Engineer at Amazon AWS in your comment.
It was very clear from their post that they were criticising STS from the perspective of an engineer in AWS within a different team.
I assumed in good faith that this is someone knowing internals as a larger customer, not an AWS person shit-talking other AWS teams.
Got curious only after a downvote hence late edit. My bad.
Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
#270Earlier quoted context omitted.
Easy to say leave, the techinical lockin cloud service providers by design choose to have makes it impossible to leave . AWS (and others) make egress costs insanely expensive for any startup to consider leaving with their data, also there is constant push to either not support open protocols or extend /expand them in ways making it hard to migrate a code base easily. If the advise is to use only effectively use manag…
> AWS (and others) make egress costs insanely expensive for any startup to consider leaving with their data I have seen this repeated many times, but don't understand it. Yes egress is expensive, but they are not THAT expensive compared to storage. S3 egress per GB is no more than 3x the price of storage, i.e. moving out just cost 3 month of storage cost (there's also API cost but that's not the one often mentioned).…
Often the other cloud vendors will assist in offering those migration costs as part of your contract negotiations.
But really, egress costs aren’t locking you in. It’s the hard coded AWS apis, terraform scripts and technical debt. Having to change all of that and refactor and reoptimize to a different providers infrastructure is a huge endeavor. That time spent night have a higher ROI being put elsewhere