Live data from Hacker News

AWS EC2/RDS Outage in us-east-1

status.aws.amazon.com

121–130 of 152 posts

Re: AWS EC2/RDS Outage in us-east-1

#121

For folks here, my RDS instances in us-east-1f are doing okay (knock on wood!) Not sure which AZ is suffering most. My client's Heroku instances are online, thankfully. Can anyone here speak to their experience with the Ohio region? I'm considering leaning on that more and more.

Your us-east-1f is not the same one as on other accounts. The letter is randomly assigned to the AZ to spread load.

Re: AWS EC2/RDS Outage in us-east-1

#122
post #110

Earlier quoted context omitted.

In a world where you have SLAs with your customers, in which you commit to something better?

Damn, these ships must really be run tightly. In every company I have worked for, the amount of outages caused by bugs and other post deployment issues was already above that number.

[deleted]

Re: AWS EC2/RDS Outage in us-east-1

#123
post #98
post #71

Earlier quoted context omitted.

For us this is exactly the correct approach. We could have spent millions of dollars and thousands of man hours hardening things to be resilient to single region outages. But for what? We aren't GE or Google. If our conference line goes down for 2-3 hours per year because we don't have apocalypse-proof infrastructure, literally nothing bad happens to our business. In this exact outage we are discussing, all of my cow…

2-3 hours per year is a lot of downtime. Most competent bare metal providers see maybe one major outage of less than an hour every 3-5 years. Nothing other than a facility wide power outage, if the load somehow gets dropped because the generators don't start right away as they should, or a misbehaving (only partially failing) core network infrastructure device should result in major outages when all the proper redund…

Is 3 hours per year really that bad? It’s 99.96% (three 9’s) uptime which I’d think is fine for most small to medium businesses.

Re: AWS EC2/RDS Outage in us-east-1

#124
post #98

Earlier quoted context omitted.

2-3 hours per year is a lot of downtime. Most competent bare metal providers see maybe one major outage of less than an hour every 3-5 years. Nothing other than a facility wide power outage, if the load somehow gets dropped because the generators don't start right away as they should, or a misbehaving (only partially failing) core network infrastructure device should result in major outages when all the proper redund…

I don't want to advertise your particular company, but if we are talking about numbers, how does your bare metal offer compares to a Amazon ec2 offer for example? And how would a customer that need to scale their load do it?

A c4 dedicated host at AWS (simply picked as the 2nd listed in the Dedicated Hosts Configuration Table as c3 the first doesn't show up under the Instance Types page) comes with an E5 2666v3, 64GB of RAM, and no storage for $810/mo.

The CPU model is non-standard, but at 10 cores and 2.90GHz is effectively a slightly higher clocked version of the E5 2660v3 (10 cores, 2.60GHz). The first google result for a 2660v3 dedicated server with an order page that allows to adjust the options (13 usable IP's, 64GB RAM, minimal 120GB SSD storage) comes out to $275.

And this is based on whole box to whole box comparison. The cost of individual instances at AWS equivalent to one of those boxes can be much higher depending on the type and size.

Re: AWS EC2/RDS Outage in us-east-1

#125
post #116

Earlier quoted context omitted.

Did you ever faced a situation where you or your clients needed more compute power and a cloud scalability would've been more convenient/cheaper?

No, a datacenter holds a seriously large amount of compute. Just a single rack is 38U usable in most cases, depending on density of compute and power availability you can get a good 2,000 CPU cores and a few dozen TiB of DDR4 from a single rack (with something like a DELL MX7000 chassis). And it's incredibly rare you'd be limited to a single rack of course. Cloud has many tangible benefits, but "amount of compute ava…

Cloud can often be cheaper than on prem or at a colo if you are both willing to be “cloud native” and change your processes and you have people who actually know what they are doing and not a bunch of “lift and shifters” who are old school netops guide, got one AWS certification and now only know how to click around in the UI and duplicate an on prem infrastructure.

Re: AWS EC2/RDS Outage in us-east-1

#126

For folks here, my RDS instances in us-east-1f are doing okay (knock on wood!) Not sure which AZ is suffering most. My client's Heroku instances are online, thankfully. Can anyone here speak to their experience with the Ohio region? I'm considering leaning on that more and more.

Your us-east-1f is not the same one as on other accounts. The letter is randomly assigned to the AZ to spread load.

Interesting, never knew that. I guess that is why the announcements never explicitly pointed to a single AZ by name.

Re: AWS EC2/RDS Outage in us-east-1

#127
post #123
post #98

Earlier quoted context omitted.

2-3 hours per year is a lot of downtime. Most competent bare metal providers see maybe one major outage of less than an hour every 3-5 years. Nothing other than a facility wide power outage, if the load somehow gets dropped because the generators don't start right away as they should, or a misbehaving (only partially failing) core network infrastructure device should result in major outages when all the proper redund…

Is 3 hours per year really that bad? It’s 99.96% (three 9’s) uptime which I’d think is fine for most small to medium businesses.

From a dedicated hosting provider's perspective, some customers will notice interruptions more than 5 minutes (1 monitoring cycle), start submitting trouble tickets at 15 minutes, and require RFO's at 30+ minutes of downtime. An hour and up even just one time, we would probably start to see cancellations. Not going to say I'm not envious that AWS seems to have a much more outage tolerant customer base.

Re: AWS EC2/RDS Outage in us-east-1

#128
post #28

The Spinnaker project is looking more appealing with every outage. Outage detected in X provider in Y region? Deploy infrastructure to Z provider in Y region.

Deploying a database with 5 TB on disk and 50k data-modifying transactions per second to multiple providers isn't exactly trivial.

Who cares about stateless, that's a solved problem.

Re: AWS EC2/RDS Outage in us-east-1

#129
post #81

Earlier quoted context omitted.

This should always be a business calculation but I do think you should note that there’s at least an order of magnitude difficulty increase between multiple AZs and regions, especially if you’re using services like RDS where it’s designed in, so I’d consider that a solid bridge step. I’m trying to put some numbers into that, I’ve been running a relatively well trafficked website in multiple AZs since 2011. We had ~20…

Good point, the multi-AZ RDS feature is a nice way to get most of the resilience upsides without any additional app complexity. You do double your database cost, but that might be worth it.

Not necessarily. You could keep the reader a smaller size and scale it up only if needed.

Re: AWS EC2/RDS Outage in us-east-1

#130
post #116

Earlier quoted context omitted.

No, a datacenter holds a seriously large amount of compute. Just a single rack is 38U usable in most cases, depending on density of compute and power availability you can get a good 2,000 CPU cores and a few dozen TiB of DDR4 from a single rack (with something like a DELL MX7000 chassis). And it's incredibly rare you'd be limited to a single rack of course. Cloud has many tangible benefits, but "amount of compute ava…

Cloud can often be cheaper than on prem or at a colo if you are both willing to be “cloud native” and change your processes and you have people who actually know what they are doing and not a bunch of “lift and shifters” who are old school netops guide, got one AWS certification and now only know how to click around in the UI and duplicate an on prem infrastructure.

Maybe if your load is unusually extremely erratic. In the vast majority of cases, you could purchase 2-3x more than you need in bare metal hosting resources (with data centre and hardware operations already outsourced), making scaling not an issue, and still see significant cost savings compared to public cloud which is typically 6-7x the cost for equivalent resources.
Post reply on HN