Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS EC2/RDS Outage in us-east-1

status.aws.amazon.com

Post reply on HN
Print view View original

141–150 of 152 posts

Re: AWS EC2/RDS Outage in us-east-1

#141

Post by mwcampbell »

hhw wrote: post #98
bob1029 wrote: post #71

Earlier quoted context omitted.

For us this is exactly the correct approach. We could have spent millions of dollars and thousands of man hours hardening things to be resilient to single region outages. But for what? We aren't GE or Google. If our conference line goes down for 2-3 hours per year because we don't have apocalypse-proof infrastructure, literally nothing bad happens to our business. In this exact outage we are discussing, all of my cow…

2-3 hours per year is a lot of downtime. Most competent bare metal providers see maybe one major outage of less than an hour every 3-5 years. Nothing other than a facility wide power outage, if the load somehow gets dropped because the generators don't start right away as they should, or a misbehaving (only partially failing) core network infrastructure device should result in major outages when all the proper redund…

> Having a code update, or some orchestration issue from your infrastructure provider be potential points of major outages are huge and unnecessary risks.

I trust any of the big cloud providers to do these things more reliably than I can. In particular, if I'm going to replicate a database across data centers within a region (availability zones as the big cloud providers call them), I'm quite sure that a managed database service will be more reliable than my own hand-configured cluster.

Re: AWS EC2/RDS Outage in us-east-1

#142

Post by karmakaze »

Dunedan wrote: post #87
karmakaze wrote: post #84

I remember reading about how not all AWS regions are similarly operated and that one was a snowflake. Is it US-East-1?

Yes. us-east-1 is the first AWS region Amazon made publicly available. It's also historically used by a lot of customers as "default region" where they launch all workloads where they don't have special needs of launching them somewhere else. That has lead to us-east-1 being the largest AWS region by far, also compromised of the largest number of availability zones (6) of all AWS regions.

Ok so it is this one. I was talking more about how the region itself has features, exceptions/quirks that are different than other AWS regions. Basically a quirks-mode region with differences that may, or may not impact you at some point in time. Or you do have special needs and US-East-1 is the only region that has the special non-standard ability you want to use.

Re: AWS EC2/RDS Outage in us-east-1

#143

Post by cthalupa »

scarface74 wrote: post #138
dijit wrote: post #137

Earlier quoted context omitted.

>I just don’t enjoy the infrastructure side that much. That's fair, that's totally your right. However, you're talking about absolute cost and unfortunately you're examples weave through true and false quite frenetically. > - we don’t want to maintain our own build servers. We just use CodeBuild with one of the prebuilt Docker containers or create a custom one. When we push our code to Github, CodeBuild brings up the…

I’m not disagreeing with you. The only times you save money by going to the cloud is by reducing the number of people you need or if you have a lot of elasticity in demand. I would never recommend anyone going to the cloud just to reproduce an infrastructure they could do at a colo. Like all things business, "want" and "cost" are different, in this case, depending on your size of course, it could easily be cheaper to…

I'll note that he says he works in the video game industry, where infra engineer/sysad/etc salaries are, in my anecdotal experience, significantly lower than even run of the mill positions in "regular" companies, and especially lower than SV/startup/big tech companies. Offers I've received from several game companies were less than 50% of what I had from other companies, and I was told when I tried to negotiate that they wanted people that were passionate about games and what they were building, and not people just looking for a cushy job. That can change the economics on the situation.

Re: AWS EC2/RDS Outage in us-east-1

#144

Post by cthalupa »

whalesalad wrote: post #136
zifnab06 wrote: post #134

Earlier quoted context omitted.

Essentially everyone picks "a" because it's the first az. There's some internal mapping to "your az a is actually datacenter q". You can kind of figure out which AZs match across accounts if you've got enough accounts you can send traffic between. I've been told that the "a" AZ you get was the least populated at object creation time (ie the first time you make an object that lives in an az), but I don't know how vali…

Aside from spinning up an EC2 node in each AZ and doing ping or tracing tests I wonder if there is a quick-n-dirty way to map AZ’s between different AWS accounts. I’ve never had to approach that scenario (cross account, low latency requirements) but in the future I’ll keep this in mind.

Go to your Subnets tab in the EC2 console. You'll see the actual AZ numbers there, vs. the 'random' lettering.

Re: AWS EC2/RDS Outage in us-east-1

#145

Post by liveoneggs »

runamok wrote: post #129
matt2000 wrote: post #100

Earlier quoted context omitted.

Good point, the multi-AZ RDS feature is a nice way to get most of the resilience upsides without any additional app complexity. You do double your database cost, but that might be worth it.

Not necessarily. You could keep the reader a smaller size and scale it up only if needed.

"dual az" is a checkbox that doubles your cost for transparent failover; it's different from the read only replica

Re: AWS EC2/RDS Outage in us-east-1

#146

Post by scarface74 »

cthalupa wrote: post #143
scarface74 wrote: post #138

Earlier quoted context omitted.

I’m not disagreeing with you. The only times you save money by going to the cloud is by reducing the number of people you need or if you have a lot of elasticity in demand. I would never recommend anyone going to the cloud just to reproduce an infrastructure they could do at a colo. Like all things business, "want" and "cost" are different, in this case, depending on your size of course, it could easily be cheaper to…

I'll note that he says he works in the video game industry, where infra engineer/sysad/etc salaries are, in my anecdotal experience, significantly lower than even run of the mill positions in "regular" companies, and especially lower than SV/startup/big tech companies. Offers I've received from several game companies were less than 50% of what I had from other companies, and I was told when I tried to negotiate that…

I’m not even coming from the perspective of a Silicon Valley big tech company. I’m in Atlanta. We would need at least three additional employees to handle our infrastructure/dev ops workload at a colo and that would be around a half a million dollars for the fully allocated cost to hire them. As opposed to the additional cost of the business support plan, the cost of the MSP, and hiring senior developers who are know their way around AWS. Also we finally hired one person to coordinate everything and take the busy work off the back of the leads.

We can do a lot with a half million dollars a year on AWS.

Re: AWS EC2/RDS Outage in us-east-1

#147

Post by neuromantik8086 »

sjwright wrote: post #62
neuromantik8086 wrote: post #56

Earlier quoted context omitted.

We wouldn't have this problem if people just used application-layer protocols and federated services like the early internet.

Wait, why wouldn’t we have these problems? Back in the 1980s, if a university campus connection goes down, you can’t telnet in or read your university POP2 email remotely. It’s down. The only difference between then and now is that we’re online (seemingly) at every waking minute expecting a hundred different services to be functional at any given moment.

Modern services such as reddit and Twitter effectively usurp the role that Usenet/NNTP and similar distributed protocols used to fulfill, but without the advantage of decentralization / lack of large single points of failure that such protocols embraced. That's what I was getting at, and maybe I'm full of shit.

In the 80s if a university campus internet connection went down, only that university was affected. Now, when a single AWS availability zone goes down, a much wider swath of users is impacted. Such consolidation / centralization shows a disregard for the spirit of the early internet and design considerations that went into it.

Again, maybe I'm full of shit. Lots of people here seem to think so.

Re: AWS EC2/RDS Outage in us-east-1

#148

Post by dijit »

scarface74 wrote: post #146
cthalupa wrote: post #143

Earlier quoted context omitted.

I'll note that he says he works in the video game industry, where infra engineer/sysad/etc salaries are, in my anecdotal experience, significantly lower than even run of the mill positions in "regular" companies, and especially lower than SV/startup/big tech companies. Offers I've received from several game companies were less than 50% of what I had from other companies, and I was told when I tried to negotiate that…

I’m not even coming from the perspective of a Silicon Valley big tech company. I’m in Atlanta. We would need at least three additional employees to handle our infrastructure/dev ops workload at a colo and that would be around a half a million dollars for the fully allocated cost to hire them. As opposed to the additional cost of the business support plan, the cost of the MSP, and hiring senior developers who are know…

To add a little context I currently spend around 500,000 CAD on infra in GCP per month, which is roughly half of my total infrastructure (in terms of raw compute/bandwidth use). The remaining metal costs 100,000 CAD / month.

As I was implying. You’re just outsourcing your ops. At scale, you end up spending significantly more than you expect.

Re: AWS EC2/RDS Outage in us-east-1

#149

Post by scarface74 »

dijit wrote: post #148
scarface74 wrote: post #146

Earlier quoted context omitted.

I’m not even coming from the perspective of a Silicon Valley big tech company. I’m in Atlanta. We would need at least three additional employees to handle our infrastructure/dev ops workload at a colo and that would be around a half a million dollars for the fully allocated cost to hire them. As opposed to the additional cost of the business support plan, the cost of the MSP, and hiring senior developers who are know…

To add a little context I currently spend around 500,000 CAD on infra in GCP per month, which is roughly half of my total infrastructure (in terms of raw compute/bandwidth use). The remaining metal costs 100,000 CAD / month. As I was implying. You’re just outsourcing your ops. At scale, you end up spending significantly more than you expect.

That’s the difference. Whether you operate at a small scale or a large scale, if you have web servers, database servers, load balancers, build servers, network infrastructure etc. If you are at a colo, you still have a minimum number of people you have to hire and no one who is any good is going to work below market rates and if they are any good, they would probably be bored out of their minds at a small company. “Outsourcing your ops” makes perfect sense until it doesn’t.

Also, when I put my software architect hat on (and take my infrastructure hat off), it’s a lot quicker to get things done just to ask our MSp to open an empty account in our AWS Organization, spin up the entire infrastructure, pilot it, get it approved, audited, and then run the same template in the production account without having to wait on a change request, approvals, pre approval security audits, etc.

I’m also not advocating all in on cloud. With a Direct Connect from your colo to your cloud infrastructure, it makes sense sometimes to have a hybrid solution. Everything from using your cloud infrastructure as a cold DR standby, using it for green field development where a team doesn’t need to be shackled by change requests, committees, etc

Re: AWS EC2/RDS Outage in us-east-1

#150

Post by dijit »

scarface74 wrote: post #149
dijit wrote: post #148

Earlier quoted context omitted.

To add a little context I currently spend around 500,000 CAD on infra in GCP per month, which is roughly half of my total infrastructure (in terms of raw compute/bandwidth use). The remaining metal costs 100,000 CAD / month. As I was implying. You’re just outsourcing your ops. At scale, you end up spending significantly more than you expect.

That’s the difference. Whether you operate at a small scale or a large scale, if you have web servers, database servers, load balancers, build servers, network infrastructure etc. If you are at a colo, you still have a minimum number of people you have to hire and no one who is any good is going to work below market rates and if they are any good, they would probably be bored out of their minds at a small company. “O…

I think we’re agreeing but we draw the line in different places.

Cloud is great for speed of deployment. But once something is made, stable and has predictable load then it’s a huge cost saving to bring it in-house. Many don’t, probably because they’ve used some cloud only technologies or fear the migration path will take time.

So you just continually line bezos pockets instead of using the cost savings to remain liquid.

Post reply on HN