Earlier quoted context omitted.
Funny thing is, the last couple interviews I've had in Chicago and Silicon Valley, I actually get points when explaining caution is necessary when using AWS for production. A magic bullet it isn't.
Whats the alternative? Building your own is certainly not.
Amazon EC2 currently down. Affecting Heroku, Reddit, Others
281–290 of 311 posts
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#282Earlier quoted context omitted.
I can't feel that Amazon are a bit of a Cassandra (mythological not the software) when these outages occur. They recommend that people failover to other availability zones but no one puts any effort into doing it then they get annoyed when a datacenter goes offline. Its not Amazons fault that you didn't make your service failure tolerant - its your fault!
I'm seeing a lot of these type of comment. The thing is, AWS completely crapped out. Don't believe their status updates that make it sound like it was a tiny little area of their data center. It was pretty much the entire zone and then whenever there is an outage affecting an entire zone it brings down global services and even other zones as well. We had servers in the bad zone and started having load issues. When I…
We didn't have downtime for various reasons but the ELBs we were using failed and the queue of starting instances was too big to see our few ones restarting.
The main systematic issue in EC2 is EBS, take that away and it will almost completely remove downtimes.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#283Earlier quoted context omitted.
To be fair, AWS downtime always make the news because they affect a lot of majors websites, but that doesn't mean an average sysadmin (or devops, whatever) would do better in term of uptime with his own bay and his toys.
But this is part of the problem: we have multiple web properties, and the fact that AWS issues can affect all of them at once is a huge downside. Certainly, if we ran on metal, we would have hardware fail, but failures would be likely to be better-isolated than at Amazon.
@override: You are hellbanned.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#284Earlier quoted context omitted.
It was a joke. I made the same joke earlier today. No-one is seriously going to believe this.
He knows you were joking. It's a bad joke. "It's a joke" is not a magic bullet that means you can do no wrong. Poisonous ideas spread as jokes. That is one of the ways they spread. A person thinking well about the issue wouldn't find the joke funny because it doesn't make sense. The joke relies on some poisonous, bad thinking to be understood. It has bad assumptions, and a bad way of looking at the world, built in.
I think it was a pretty good joke, it's funny specifically because amazon obviously won't be doing it deliberately, no poisonous thinking required.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#285Earlier quoted context omitted.
Whats the alternative? Building your own is certainly not.
Why not? http://blogs.technet.com/b/privatecloud/archive/2012/03/12/l...
Well the reason why seed money is so low these days is because people expect you to not spend all the money on making your own cloud.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#286Earlier quoted context omitted.
I can't feel that Amazon are a bit of a Cassandra (mythological not the software) when these outages occur. They recommend that people failover to other availability zones but no one puts any effort into doing it then they get annoyed when a datacenter goes offline. Its not Amazons fault that you didn't make your service failure tolerant - its your fault!
perhaps they should make it easier to do so, as in having some default option you could select, at a premium of course.
Perhaps they should - though Rightscale.com provides a service that helps you do just that. (Disclaimer: I do not work for Rightscale or know anyone that works there)
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#287Earlier quoted context omitted.
Funny thing is, the last couple interviews I've had in Chicago and Silicon Valley, I actually get points when explaining caution is necessary when using AWS for production. A magic bullet it isn't.
I can't feel that Amazon are a bit of a Cassandra (mythological not the software) when these outages occur. They recommend that people failover to other availability zones but no one puts any effort into doing it then they get annoyed when a datacenter goes offline. Its not Amazons fault that you didn't make your service failure tolerant - its your fault!
Right. Blame it on the victim. How do you make a "fault tolerant" service when core services like ELB together with the API behind it start to fail? Multi-region? Multi-cloud? When is it "designed to make web-scale computing easier" part supposed to kick in? With half-baked producs like ELB or things like EIP that cease to work when you need them the most?
I actually asked the AWS Premium support regarding the ELB multi-AZ issues, in order to actually make things easier for everyone. This is the answer I got:
"As it stands right now, you would need to make a call to ELB to disable the failed AZ. It may be possible for you to programatically/script this process in the case of an event.
Going forward, this is something that we would like to address but I don't have any ETA for when something like this might be implemented."
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#288Earlier quoted context omitted.
I'm seeing a lot of these type of comment. The thing is, AWS completely crapped out. Don't believe their status updates that make it sound like it was a tiny little area of their data center. It was pretty much the entire zone and then whenever there is an outage affecting an entire zone it brings down global services and even other zones as well. We had servers in the bad zone and started having load issues. When I…
We didn't have downtime for various reasons but the ELBs we were using failed and the queue of starting instances was too big to see our few ones restarting. The main systematic issue in EC2 is EBS, take that away and it will almost completely remove downtimes.
The problem with ELBs is that they are themselves EC2 instances and use many of the global services for detecting load, scaling up, etc.. Like all of AWS's value-added services, they are therefore more likely to fail during an outage event, not less likely, as they depend on more services.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#289For added fun, their EC2 console is down. I got this for a while: Http/1.1 Service Unavailable ... then an empty console saying "loading" for the last 20 minutes. Then recently it upgraded to saying "Request limit exceeded." in place of the loading message (because hey, I'd refreshed the page four times over the course of 20 minutes). On the upside, their status page shows all green lights.
They've acknowledged that at http://status.aws.amazon.com/ for awhile now with a tiny "i" status icon (I can't load my instances pane): 12:07 PM PDT We are experiencing elevated error rates with the EC2 Management Console.
Amazon misrepresents reality on this status page!
They have standardized icons to represent various levels of issues (orange = perf issue, red = disruption). But they don't even use them. Instead they add [i] to the green icons to indicate perf issues (Amazon Elastic Compute Cloud - N. Virginia) and disruptions of service (Amazon Relational Database Service - N. Virginia).
Maybe this status page is controlled by marketing bozos who want to pretend the situation is not so bad.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#290I am frankly surprised so many still use EC2 considering how frequently it breaks. It's not cheap, so the only reasons to use it would be reliability or scale right? Why not just get lots of boxes at Hetzner and OVH (40 EUR a month for 32GB of RAM and 4x2x3.4GHz cores) and scale up / redundansize that way?