Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS EC2/RDS Outage in us-east-1

status.aws.amazon.com

Post reply on HN
Print view View original

131–140 of 152 posts

Re: AWS EC2/RDS Outage in us-east-1

#131

Post by scarface74 »

tus88 wrote: post #117
matt2000 wrote: post #61

Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…

Or how about no AZ where possible using a serverless architecture/lambda?

“Serverless” is not magic. The minute you need to attach to your VPC (I refuse to say “run inside your VPC, that’s not correct), you still have to worry about multi AZ. If you are using RDS, as opposed to DynamoDB, you have to configure it for multi AZ.

I use lambda all of the time and I’m definitely not afraid of the “lock in” boogeyman, but I always architect my lambda’s to make moving away from lambda to either Fargate or just an EC2 instance as easy as possible.

On another note, it’s just as easy to architect your regular old EC2 instances running stateless servers to be AZ failure resilient. Just set up an autoscaling group with a min/max of 1 and configure it to work across multiple AZ’s.

Also lambda comes with its own set of limitation - maximum runtimes of 15 minutes, cold start times, temporary storage space of only a half a gig, limited CPU/memory options, etc.

Re: AWS EC2/RDS Outage in us-east-1

#133

Post by scarface74 »

hhw wrote: post #130
scarface74 wrote: post #125

Earlier quoted context omitted.

Cloud can often be cheaper than on prem or at a colo if you are both willing to be “cloud native” and change your processes and you have people who actually know what they are doing and not a bunch of “lift and shifters” who are old school netops guide, got one AWS certification and now only know how to click around in the UI and duplicate an on prem infrastructure.

Maybe if your load is unusually extremely erratic. In the vast majority of cases, you could purchase 2-3x more than you need in bare metal hosting resources (with data centre and hardware operations already outsourced), making scaling not an issue, and still see significant cost savings compared to public cloud which is typically 6-7x the cost for equivalent resources.

If all you care about is running a bunch of VMs yes. But if you’re just running a bunch of VMs and using cloud hosting - you’re doing it wrong.

- we don’t want to maintain our own build servers. We just use CodeBuild with one of the prebuilt Docker containers or create a custom one. When we push our code to Github, CodeBuild brings up the Docker container, runs the build and the unit tests and puts the artifacts on S3.

- We don’t want to maintain our own messaging, sftp, database, scheduler, load balancer, oath, object storage, servers etc. We don’t have to, AWS does that.

- We don’t want to manage our own app and web servers. We just use Lambda and Fargate and give AWS a zip file/docker container and it runs it for us.

- We need to run a temporary stress test environment or want to do a quick proof of concept, we create s Cloud Formation template, spin up the entire environment, run our tests with different configurations and kill it. When we want to spin it up again, we just run the template.

We don’t have to pay for a staff of people to babysit our infrastructure between our business support contract with AWS and the MSP, we can get all of the troubleshooting and busy work support as needed.

I’m a software engineer/architect by title, but if you look at my resume from another angle, I would be qualified to be an AWS architect. I just don’t enjoy the infrastructure side that much.

Re: AWS EC2/RDS Outage in us-east-1

#134

Post by zifnab06 »

whalesalad wrote: post #126
deusex_ wrote: post #121

Earlier quoted context omitted.

Your us-east-1f is not the same one as on other accounts. The letter is randomly assigned to the AZ to spread load.

Interesting, never knew that. I guess that is why the announcements never explicitly pointed to a single AZ by name.

Essentially everyone picks "a" because it's the first az. There's some internal mapping to "your az a is actually datacenter q". You can kind of figure out which AZs match across accounts if you've got enough accounts you can send traffic between.

I've been told that the "a" AZ you get was the least populated at object creation time (ie the first time you make an object that lives in an az), but I don't know how valid that is.

Re: AWS EC2/RDS Outage in us-east-1

#135

Post by unixhero »

freehunter wrote: post #90
manojlds wrote: post #75

Earlier quoted context omitted.

Being on AWS is also easy to explain to customers about downtimes - AWS was down and customers are pretty understanding in that case and don't demand why you aren't multi AZ etc ( of course YMMV based on sensitivity of your business)

Kind of like the "no one ever got fired for recommending IBM", if you have significant downtime on Linode or Hetzner, people are going to ask "why weren't you on AWS?!". If you're on AWS and AWS goes down, you get to skip that question entirely. You were already using the logical choice, you don't need to defend anything. If Netflix can go down when AWS goes down, so can your app. AWS outages impact so much of the In…

Great point.

Re: AWS EC2/RDS Outage in us-east-1

#136

Post by whalesalad »

zifnab06 wrote: post #134
whalesalad wrote: post #126

Earlier quoted context omitted.

Interesting, never knew that. I guess that is why the announcements never explicitly pointed to a single AZ by name.

Essentially everyone picks "a" because it's the first az. There's some internal mapping to "your az a is actually datacenter q". You can kind of figure out which AZs match across accounts if you've got enough accounts you can send traffic between. I've been told that the "a" AZ you get was the least populated at object creation time (ie the first time you make an object that lives in an az), but I don't know how vali…

Aside from spinning up an EC2 node in each AZ and doing ping or tracing tests I wonder if there is a quick-n-dirty way to map AZ’s between different AWS accounts. I’ve never had to approach that scenario (cross account, low latency requirements) but in the future I’ll keep this in mind.

Re: AWS EC2/RDS Outage in us-east-1

#137

Post by dijit »

scarface74 wrote: post #133
hhw wrote: post #130

Earlier quoted context omitted.

Maybe if your load is unusually extremely erratic. In the vast majority of cases, you could purchase 2-3x more than you need in bare metal hosting resources (with data centre and hardware operations already outsourced), making scaling not an issue, and still see significant cost savings compared to public cloud which is typically 6-7x the cost for equivalent resources.

If all you care about is running a bunch of VMs yes. But if you’re just running a bunch of VMs and using cloud hosting - you’re doing it wrong. - we don’t want to maintain our own build servers. We just use CodeBuild with one of the prebuilt Docker containers or create a custom one. When we push our code to Github, CodeBuild brings up the Docker container, runs the build and the unit tests and puts the artifacts on S…

>I just don’t enjoy the infrastructure side that much.

That's fair, that's totally your right.

However, you're talking about absolute cost and unfortunately you're examples weave through true and false quite frenetically.

> - we don’t want to maintain our own build servers. We just use CodeBuild with one of the prebuilt Docker containers or create a custom one. When we push our code to Github, CodeBuild brings up the Docker container, runs the build and the unit tests and puts the artifacts on S3.

Like all things business, "want" and "cost" are different, in this case, depending on your size of course, it could easily be cheaper to have a dedicated "build engineer" maintaining a build farm. This is how the majority of people do it. (I work in the video games industry, it's _MUCH_ cheaper to do it this way for us)

> - We don’t want to maintain our own messaging, sftp, database, scheduler, load balancer, oath, object storage, servers etc. We don’t have to, AWS does that.

Again, those are "wants", TCO can be much lower when out of the cloud. But again, depends on scale. (as in, lower scale is cheaper on cloud, not larger scale).

> - We don’t want to manage our own app and web servers. We just use Lambda and Fargate and give AWS a zip file/docker container and it runs it for us.

I mean, 1 sysadmin can automate/orchestrate literally thousands of webservers.

>- We need to run a temporary stress test environment or want to do a quick proof of concept, we create s Cloud Formation template, spin up the entire environment, run our tests with different configurations and kill it. When we want to spin it up again, we just run the template.

Yes, this is a real strength of cloud.

> We don’t have to pay for a staff of people to babysit our infrastructure between our business support contract with AWS and the MSP, we can get all of the troubleshooting and busy work support as needed.

Yes, but you are paying "overhead" for all of that, and not having talented engineers on your payroll who understand your business critical systems is, in my opinion, foolish.

I've dealt with vendor support and it's incredibly hit and miss, and it's much more "miss" when you're a smaller customer to the vendor. Of course, this is anecdotal.

Re: AWS EC2/RDS Outage in us-east-1

#138

Post by scarface74 »

dijit wrote: post #137
scarface74 wrote: post #133

Earlier quoted context omitted.

If all you care about is running a bunch of VMs yes. But if you’re just running a bunch of VMs and using cloud hosting - you’re doing it wrong. - we don’t want to maintain our own build servers. We just use CodeBuild with one of the prebuilt Docker containers or create a custom one. When we push our code to Github, CodeBuild brings up the Docker container, runs the build and the unit tests and puts the artifacts on S…

>I just don’t enjoy the infrastructure side that much. That's fair, that's totally your right. However, you're talking about absolute cost and unfortunately you're examples weave through true and false quite frenetically. > - we don’t want to maintain our own build servers. We just use CodeBuild with one of the prebuilt Docker containers or create a custom one. When we push our code to Github, CodeBuild brings up the…

I’m not disagreeing with you. The only times you save money by going to the cloud is by reducing the number of people you need or if you have a lot of elasticity in demand. I would never recommend anyone going to the cloud just to reproduce an infrastructure they could do at a colo.

Like all things business, "want" and "cost" are different, in this case, depending on your size of course, it could easily be cheaper to have a dedicated "build engineer" maintaining a build farm. This is how the majority of people do it. (I work in the video games industry, it's _MUCH_ cheaper to do it this way for us)

That’s $80K to $100K. You can buy a lot on AWS for that price...

Again, those are "wants", TCO can be much lower when out of the cloud. But again, depends on scale. (as in, lower scale is cheaper on cloud, not larger scale).

That’s another $100K to $200K....

I mean, 1 sysadmin can automate/orchestrate literally thousands of webservers.

That’s yet another $100K. You’re up to at least $250K - $500K In salaries.

Yes, but you are paying "overhead" for all of that, and not having talented engineers on your payroll who understand your business critical systems is, in my opinion, foolish.

I am one of the “talented engineers” that’s why I mentioned I could go out and get a job tomorrow as an AWS architect - my resume is very much buzzword compliant with what it would take from both a development, Devops, and netops side to hold my own in a small to medium size company. I just find that side of the fence boring - we outsource the boring work or the “undifferentiated heavy lifting”.

As that side got too much for me, we hired one dedicated sysadmin type person to coordinate between what he would do himself, our clients and our MSP.

I’m actually trotted our as the “infrastructure architect” to our clients even though my official title and day to day work is a developer. I haven’t embarrassed us yet.

I've dealt with vendor support and it's incredibly hit and miss, and it's much more "miss" when you're a smaller customer to the vendor. Of course, this is anecdotal.

I agree completely. If it’s something complex, I either do it myself or have very detailed requirements on what our needs are. But honestly, the more managed services you use, the less you have to do that part.

Re: AWS EC2/RDS Outage in us-east-1

#139

Post by austinshea »

matt2000 wrote: post #61

Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…

I can understand if the networking implications and data replication issues are too complicated for you, but, if you have failover in the same region, you’re already paying the extra cost.

Working with multiple regions is cost friendly on AWS. You should put in the time and learn how that stuff is done, it’s not as complicated as you think.

Re: AWS EC2/RDS Outage in us-east-1

#140

Post by acdha »

runamok wrote: post #129
matt2000 wrote: post #100

Earlier quoted context omitted.

Good point, the multi-AZ RDS feature is a nice way to get most of the resilience upsides without any additional app complexity. You do double your database cost, but that might be worth it.

Not necessarily. You could keep the reader a smaller size and scale it up only if needed.

That’s assuming that you a) religiously test with the smaller size and b) are comfortable that scaling up will work when lots of other people are shifting workloads, too. I usually work on projects where we haven’t wanted to deal with that but that’s a judgement call.
Post reply on HN