111–120 of 152 posts
Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…
For us this is exactly the correct approach. We could have spent millions of dollars and thousands of man hours hardening things to be resilient to single region outages. But for what? We aren't GE or Google. If our conference line goes down for 2-3 hours per year because we don't have apocalypse-proof infrastructure, literally nothing bad happens to our business. In this exact outage we are discussing, all of my cow…
IMO you should do everything you can as multi AZ and if there are some services that are harder to do and you don't need it, then put them in a single AZ.
Thing is that if you keep everything in a single AZ it will be much harder to change when this requirement become important.
Earlier quoted context omitted.
Being on AWS is also easy to explain to customers about downtimes - AWS was down and customers are pretty understanding in that case and don't demand why you aren't multi AZ etc ( of course YMMV based on sensitivity of your business)
Yep. I call it the "AWS Chicken Pox Party." We got lucky this time, RDS, ELB, EC2 and Lightsail instances all in US-East-1 across multiple accounts and no issues (knock-on-wood and understand that we'll get it the next time). Especially happy as I had a two-day running neural net training job running and it's still going, that would have been depressing. Phew.
Earlier quoted context omitted.
2-3 hours per year is a lot of downtime. Most competent bare metal providers see maybe one major outage of less than an hour every 3-5 years. Nothing other than a facility wide power outage, if the load somehow gets dropped because the generators don't start right away as they should, or a misbehaving (only partially failing) core network infrastructure device should result in major outages when all the proper redund…
Don’t mind me. I’m just here to corroborate your claims of downtime as a consumer of bare-metal hosting providers for 15-something years.
> 10:47 AM PDT We want to give you more information on progress at this point, and what we know about the event. At 4:33 AM PDT one of 10 datacenters in one of the 6 Availability Zones in the US-EAST-1 Region saw a failure of utility power. Backup generators came online immediately, but for reasons we are still investigating, began quickly failing at around 6:00 AM PDT. This resulted in 7.5% of all instances in that Availability Zone failing by 6:10 AM PDT. Over the last few hours we have recovered most instances but still have 1.5% of the instances in that Availability Zone remaining to be recovered. Similar impact existed to EBS and we continue to recover volumes within EBS. New instance launches in this zone continue to work without issue.
Earlier quoted context omitted.
Don’t mind me. I’m just here to corroborate your claims of downtime as a consumer of bare-metal hosting providers for 15-something years.
Did you ever faced a situation where you or your clients needed more compute power and a cloud scalability would've been more convenient/cheaper?
And it's incredibly rare you'd be limited to a single rack of course.
Cloud has many tangible benefits, but "amount of compute available" is not one of them. Time to acquire compute, though, is. (and, obviously, management of the resources/datacenter operations).
Cloud is almost never cheaper, even factoring in salaries. It's just very convenient if you're small enough not to have people providing compute very well internally. (and, internally, people tend to understaff/underfund the teams that would do the same job as cloud operators are doing)
Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…
Earlier quoted context omitted.
Yep. I call it the "AWS Chicken Pox Party." We got lucky this time, RDS, ELB, EC2 and Lightsail instances all in US-East-1 across multiple accounts and no issues (knock-on-wood and understand that we'll get it the next time). Especially happy as I had a two-day running neural net training job running and it's still going, that would have been depressing. Phew.
Why don't you just checkpoint the model every n steps? NNs fail for a myriad of reasons, you can easily reduce risk by routinely saving state.
Earlier quoted context omitted.
3 hours of downtime per year equals to 99,96% uptime. In what world is that a lot of downtime?
In a world where you have SLAs with your customers, in which you commit to something better?
In every company I have worked for, the amount of outages caused by bugs and other post deployment issues was already above that number.
Earlier quoted context omitted.
2-3 hours per year is a lot of downtime. Most competent bare metal providers see maybe one major outage of less than an hour every 3-5 years. Nothing other than a facility wide power outage, if the load somehow gets dropped because the generators don't start right away as they should, or a misbehaving (only partially failing) core network infrastructure device should result in major outages when all the proper redund…
3 hours of downtime per year equals to 99,96% uptime. In what world is that a lot of downtime?
Usually you strive for "five 9's" in infrastructure, obviously there's a lot of wiggle room depending on business case. But reliability for individual components gets exponentially harder with each 9 after the first 2.
99.96% uptime of a datacenter is shockingly low, taking connection issues into account (IE; number of successful inbound packets vs unsuccessful ones, not just served requests). For context my company has around 15 datacenters around the world which routinely hit 5-9's, with only a few issues of datacenters being down for 2-3 minutes during a particularly bad ISP outage.
The overwhelming majority of degradations are ones related to bad code being deployed. But since reliability is a sum of all components availability it follows that permitting more outages is less preferable. Especially since they affect all or at least the majority of components in a given region.