Ruh Roh. The service I'm using to acquire accommodation seems to be dependent on AWS. Guess I'm going to be homeless tomorrow if it doesn't get fixed. :X
Note to self. Don't ever build a service reliant on AWS.
Amazon Web Services are down
111–120 of 346 posts
Re: Amazon Web Services are down
#112I guess this is one Reddit outage that can't be blamed on poor scaling
I was to be under the impression a great deal of Reddits issues were linked to Amazon.
Re: Amazon Web Services are down
#113Earlier quoted context omitted.
I was looking at EC2 until this! I thought you could cluster your instances across many regions and replicate blah blah blah and change your elastic ip addresses in instances like this? Is this a case that it's not being utilised or does that system not work? I appreciate you are busy right now so I'm not expecting a reply any time soon.
That is the theory, but all of our data is currently locked in the inaccessible EBS system. I'd still say Amazon is a great place for your startup, just don't use EBS.
Or have a fallback, maybe?
Re: Amazon Web Services are down
#114Earlier quoted context omitted.
You said 6 words in your original post. And now you're trying to suggest I was to get the above 166 words from the original 6? You raise perfectly valid and interesting points. It's just a shame they're not the parent comment because then they would have invoked some thoughtful discussion.
It's a shame your sarcastic reply derailed the conversation.
Fair call.
Re: Amazon Web Services are down
#115Assuming the problem is indeed with EBS, I would say this should be a warning sign to anyone considering going with a PaaS provider, which Amazon is quickly becoming, instead of an IaaS provider like Slicehost or Linode. The increased complexity of their offering makes it more likely that things will break, leaving you locked in. I did a 15 minute talk on the subject, which you can check out here: http://iforum.com.u…
Every time someone makes the claim that downtime should be a warning sign about going with a PaaS provider (or, indeed, an IaaS provider, or in some cases, people even make this claim about going with someone else's Data Center) - I always respond:
"And why do you believe that you would do any better?"
Every environment I've been involved in as an operations professional for the last 15 years has experienced downtime regardless of how much we invested in staging, testing, change control procedures, redundancy and ITIL methodology.
While it is true that there are some environments in the world that experience no, or close to no downtime (The NYSE, google.com, the Space Shuttle) - it's usually not worth the investments those organizations make in ensuring 100% uptime.
I'm more interested in seeing what the monitoring methodology is, what the response protocols are, and what the reported downtime has been over the last year has been, than I am about getting too concerned about the occasional outage.
Re: Amazon Web Services are down
#116Re: Amazon Web Services are down
#117Assuming the problem is indeed with EBS, I would say this should be a warning sign to anyone considering going with a PaaS provider, which Amazon is quickly becoming, instead of an IaaS provider like Slicehost or Linode. The increased complexity of their offering makes it more likely that things will break, leaving you locked in. I did a 15 minute talk on the subject, which you can check out here: http://iforum.com.u…
I dont see EBS as a very PaaS type service. It is a virtualised network storage volume with snapshotting, just what you might expect from an IaaS provider. Clearly it is one of the more complex parts of the Amazom infrastructure though (other providers dont provide anything similar). If you dont use EBS many applications would have to implement a lot of its functionality themselves, and I suspect this would lead to some data loss instead, when drives fail or instances fail leaving drives unavailable.
You can avoid EBS if you use local drives and s3, it is just difficult to run a database like this if it was not built that way.
Re: Amazon Web Services are down
#118Earlier quoted context omitted.
I was looking at EC2 until this! I thought you could cluster your instances across many regions and replicate blah blah blah and change your elastic ip addresses in instances like this? Is this a case that it's not being utilised or does that system not work? I appreciate you are busy right now so I'm not expecting a reply any time soon.
That is the theory, but all of our data is currently locked in the inaccessible EBS system. I'd still say Amazon is a great place for your startup, just don't use EBS.
I thought you could snapshot drives across regions and bring those EBS drives up under new instances in new regions?
I have not used it all in detail yet so I don't know the practicality of this method.
I think I will stick to my co-location costings I am doing for the time being. There is only one person to rely on when it all goes wrong then!
Good luck getting it sorted, i know I wouldn't appreciate being up at 3am sorting it though
Re: Amazon Web Services are down
#119Earlier quoted context omitted.
This outage is affecting all AZ's in the East. So even a multizone setup wouldn't help for this one. Only a multiregion setup. This outage is a lot like having your entire datacenter lose power.
If "multi-region" means North America, Europe and Asia Pacific, doing so would also improve world-wide latency (e.g. here in Australia...). Could you use this outage to justify switching to multi-region?
A blog post last month touched on this:
"Q: Why is reddit tied so tightly to the affected availability zone?
A: When we started with Amazon, our code was written with the assumption that there would be one data center. We have been working towards fixing this since we moved two years ago. Unfortunately, progress has been slow in this area. Luckily, we are currently in a hiring round which will increase the technical staff by 200% :) These new programmers will help us address this issue."
Not sure if the costs of data transfer between regions (charged at full internet price) would justify the added reliability/lower latency though.
Re: Amazon Web Services are down
#120Earlier quoted context omitted.
Don't worry. If skynet is in EC2, we'll be fine.
You don't understand, Skynet is using all Amazon resources, hence the outages ;-) Amazon have stated many times that amazon.com itself runs mostly on AWS platform, but it works fine now ...
AWS platform on a private cloud, it is not the same as AWS platform for us commoners.