11–20 of 48 posts
Everything fails. Design systems that minimize the impact failures have on your customers. Moving to another provider isn't going to fix the problem. Data like this should be stored in more than one place.
In reality, replicating across two clouds is difficult and expensive. It's quite possible that Bitbucket wouldn't exist at all if they had used such an architecture.
Dear HN readers: What according to you could be one way of architecting the storage so as to avoid similar AWS EBS outages in the future?
Eventually you architect your application to distribute load over multiple facilities and to become resilient against component failure.
Until then: You do nothing, grab a beer, relax and wait as the amazon guys sweat their asses off to fix it. You pat yourself on the back because it is not your ass in the trenches right now.
On top of that you have the perfect excuse for the followup blog-post.
We use EBS extensively on our infrastructure. Occasionally an EBS volume will fail. We ran into an issue where a volume had a spike in IO load as I think was the case here.
EBS volumes are not magic, they are just chunks of physical disks. Disks fail. You should have the system architected so that you can handle such failures.
Here are a few things you can do (we don't do all of these, but enough to assure we won't lose data or have downtime in the case of a failure.)
- Mount several EBS volumes, use raid.
- If its a database, set up a failover node on a separate EBS volume.
- Take regular snapshot backups.
- Take regular full backups to S3.
It's also very important to have everything highly automated. If an EBS volume fails for us, its one command to switch to the failover node. If that doesn't work, its another command to spin up a new machine off the last available snapshot with a few hours of data loss (worst case scenario.) Everything is highly monitored with nagios + ganglia so we know when bad stuff happens.
The two or three times we've had issues with EBS we were able to either switch to a failover node or take a snapshot and mount a new volume from there. I haven't set up RAID on EC2, but I'd imagine this also a very good route to protect your data.
Remember, the cloud isn't magic. The only advantage you get with the cloud is rapid provisioning and unlimited capacity if you need it. You still have to build a shared nothing, reliable architecture within the framework the cloud gives you. We've found EC2 and EBS to work out very well, but of course there were growing pains as you learn very quickly where your single points of failure are! I get the sense that the overall reliability of resources such as instances or volumes, on the whole, is definitely lower than what you'd expect in a standard hosting provider, whatever the reason may be.
Edit: Of course, you could also load your data on a distributed data store like Cassandra as well that handles some of this failover and replication magic automatically, too.
An unfortunate event, however, re-iterates to system admins why the cloud should only be used as a low tier of storage -- for now.
Everything fails occassionally. Amazon probably has a team of highly specialized engineers on the task right now, working under the pressure of a few dozen disgruntled customers and under the eyes of worldwide press.
Could your company respond with an equal intensity if this was your own hardware? Will your SAN supplier whip his staff on-site as fast as they will for amazon?
I'm here to answer questions if there are any (I run Bitbucket.)
Unfortunately this doesn't sound like an EBS issue but a systems architecture flaw. I apologize in advance if my analysis is wrong here, but I think it's important to understand. We use EBS extensively on our infrastructure. Occasionally an EBS volume will fail. We ran into an issue where a volume had a spike in IO load as I think was the case here. EBS volumes are not magic, they are just chunks of physical disks. D…
"Each storage volume is automatically replicated within the same Availability Zone. This prevents data loss due to failure of any single hardware component."
We don't keep the database on the same EBS, and we have segmented database traffic out to several EBS volumes (for WAL, etc.) That's not the issue.
We take regular snapshot backups. We didn't lose any data. We have everything, we just can't get to it.
Regardless of what might make sense in this situation, it's not working for us. We've moved both our instances and the volumes to different availability zones, to no avail.
I just received a call from AWS engineering, assuring us that we are currently their top priority, and a team of engineers are working to fix the problem. They're seeing the issue on their end, and fortunately for them, it seems rather isolated to our instance.
Could we have taken precautions to prevent this problem? Maybe. We hadn't, cause we didn't anticipate a problem as exotic as this one. The only way to keep persistent data on EC2 is using EBS, and right now, it doesn't work for us, at all. This is not a common problem that could've been solved with backups or snapshots, or whatever.
An unfortunate event, however, re-iterates to system admins why the cloud should only be used as a low tier of storage -- for now.
What you are saying makes no sense. Everything fails occassionally. Amazon probably has a team of highly specialized engineers on the task right now, working under the pressure of a few dozen disgruntled customers and under the eyes of worldwide press. Could your company respond with an equal intensity if this was your own hardware? Will your SAN supplier whip his staff on-site as fast as they will for amazon?
I wish I could use that line to explain to my company why our business has come to a complete halt.
Granted, your business is completely web-based and in the cloud, which is why I specifically made a mention to system admins why the cloud is not reliable enough to be a high-level tier of storage yet. Why doesn't that make sense? I wasn't trying to offend you or your decisions.
Also, yes, my company and my SAN supplier would have staff on-site. But we have control over our own hardware, so there's really no comparison.
I'm here to answer questions if there are any (I run Bitbucket.)
What do you think of the other post critcizing your architecture? I am doing my own arch work on EC2, so I am trying to understand exactly what caused this failure. Is this a problem with your instance being able to access any EBS? Why couldn't you spin up another instance with a fresh EBS from a backup and redirect DNS to that instance?
You can't anticipate everything, and as I've pointed out in another comment here, this one is rather exotic.
Quick summary of what the problem is: We have an EBS volume. It mounts fine, appears fine. The problem is that it's excruciatingly slow. We can't serve data from the volume at any speed, really. Running an "ls" takes over a minute, in a small directory.
All systems are running, everything should be fine, but seeing as we can't read the data fast enough, we've been forced to put a static page explaining what's going on.
Booting a new instance, re-creating the volume from a recent snapshot, doesn't help. The exact same problem persists. Why? We don't know. Amazon's figuring it out.
We're doing everything we can do remedy the problem, but unfortunately right now, that consists of our team drinking coffee to not fall asleep, waiting for the final call from Amazon telling us they've sorted it out.
Unfortunately this doesn't sound like an EBS issue but a systems architecture flaw. I apologize in advance if my analysis is wrong here, but I think it's important to understand. We use EBS extensively on our infrastructure. Occasionally an EBS volume will fail. We ran into an issue where a volume had a spike in IO load as I think was the case here. EBS volumes are not magic, they are just chunks of physical disks. D…
You're absolutely right that the cloud is not magic, but you do get some guarantees with EBS. From their website: "Each storage volume is automatically replicated within the same Availability Zone. This prevents data loss due to failure of any single hardware component." We don't keep the database on the same EBS, and we have segmented database traffic out to several EBS volumes (for WAL, etc.) That's not the issue.…
S3 should work too? Unless it was a global EBS failure, you should be able to restore from any backup to a new set of instances and stores, why doesn't that work?