Reddit is down super-often for me. This very minute, I can't log in or use the site logged in, it just serves me 503 errors.
Why did we take reddit down for 71 minutes?
11–20 of 47 posts
Re: Why did we take reddit down for 71 minutes?
#12Earlier quoted context omitted.
I think it works like this: Even if Amazon uses 'k' hdds for one EBS "disk", since you're sharing the real hdds with other users, you don't get 'k' hdds' performance, you only get a fraction. By RAIDing over 'n' EBS "disks", you are effectively compensating for the reduced performance due to sharing.
I get what the stack looks like, but it seems really broken and likely quite inefficient. Thus far, Amazon has gone after greenfield applications which can be written within the constraints of their cloud platform. However, there are a ton of people hosting their own SQL database-based apps where a single DB is the bottleneck. Without significant refactoring, these apps can only scale vertically with the DB. So, whil…
An EBS volume has the performance of a ~10-disk RAID; it's hardly desktop class. It would be nice if they offered wide-striped volumes, but Amazon's strategy is to not do anything that customers can kludge for themselves.
Re: Why did we take reddit down for 71 minutes?
#13Earlier quoted context omitted.
I think it works like this: Even if Amazon uses 'k' hdds for one EBS "disk", since you're sharing the real hdds with other users, you don't get 'k' hdds' performance, you only get a fraction. By RAIDing over 'n' EBS "disks", you are effectively compensating for the reduced performance due to sharing.
I get what the stack looks like, but it seems really broken and likely quite inefficient. Thus far, Amazon has gone after greenfield applications which can be written within the constraints of their cloud platform. However, there are a ton of people hosting their own SQL database-based apps where a single DB is the bottleneck. Without significant refactoring, these apps can only scale vertically with the DB. So, whil…
You hit the nail on the head as to why I'm looking into a physical DB server with RAIDed SSD's instead of hopping onto EC2. I would love to use Amazon and not have to deal with the potential headaches of managing physical machines, but the stories (maybe FUD) of having to raid EBS instances, spool up 20 instances to find the winners and kill the rest, etc etc really kills the appeal.
If they could promise me consistent database performance on par with a really nice physical machine, I would gladly fork over 500/month for it.
Re: Why did we take reddit down for 71 minutes?
#14One major reason not to hop on the cloud bandwagon just yet is issues like these. The more layers underneath that are not under your control the more layers you'll have to add to remedy that.
Systems with excessive complexity are hard to debug, especially when it comes to analyzing performance issues.
Given complete control of the hardware from the ground up it can already be quite hard to accurately pinpoint a bottle neck so you can solve it. Adding a lot of stuff between your code and the hardware is not going to make that any easier.
Typically a stack has 6 layers before you get to your application: drive, controller, driver, filesystem, database, app.
In a cloud environment anything under the filesystem layer is effectively out of your control and out of your ability to troubleshoot. The solution, to add another layer of complexity in order to combat the slowdown is really the opposite of what an ideal cloud environment would give you.
After all, the #1 selling point of the cloud is scalability and performance.
I think that it would be best if Amazon worked together with the OP to resolve the issue as a problem ticket rather than to try to solve it by adding a software raid.
Of course, that's just armchair reasoning, not being in the hot seat makes life easier.
Re: Why did we take reddit down for 71 minutes?
#15One major reason not to hop on the cloud bandwagon just yet is issues like these. The more layers underneath that are not under your control the more layers you'll have to add to remedy that. Systems with excessive complexity are hard to debug, especially when it comes to analyzing performance issues. Given complete control of the hardware from the ground up it can already be quite hard to accurately pinpoint a bottl…
After all, the #1 selling point of the cloud is scalability and performance.
I think you're missing the key component that it's scalability and performance that you don't have to manage yourself. And, yes, when you don't manage it yourself, it's going to be much more difficult to diagnose performance problems. But someone else managing the infrastructure also means that people who would be incapable of diagnosing the problem anyway (either from lack of expertise or lack of time) have access to the resources.
Re: Why did we take reddit down for 71 minutes?
#16One major reason not to hop on the cloud bandwagon just yet is issues like these. The more layers underneath that are not under your control the more layers you'll have to add to remedy that. Systems with excessive complexity are hard to debug, especially when it comes to analyzing performance issues. Given complete control of the hardware from the ground up it can already be quite hard to accurately pinpoint a bottl…
If they had a dedicated server stack, they would probably have done the same thing as a hardware RAID anyway. They just replicated in software what they would have done in hardware.
Re: Why did we take reddit down for 71 minutes?
#17Does anyone know if reddit is doing anything special to create the RAID? Or is it just mdadm?
Re: Why did we take reddit down for 71 minutes?
#18One major reason not to hop on the cloud bandwagon just yet is issues like these. The more layers underneath that are not under your control the more layers you'll have to add to remedy that. Systems with excessive complexity are hard to debug, especially when it comes to analyzing performance issues. Given complete control of the hardware from the ground up it can already be quite hard to accurately pinpoint a bottl…
After all, the #1 selling point of the cloud is scalability and performance. I think you're missing the key component that it's scalability and performance that you don't have to manage yourself. And, yes, when you don't manage it yourself, it's going to be much more difficult to diagnose performance problems. But someone else managing the infrastructure also means that people who would be incapable of diagnosing the…
It definitely looks as though they ended up having to manage it themselves.
Basically this seems to put a fairly low upper limit to using 'the cloud' for something a little larger before you get back exactly the same kinds of issues that you were dealing with when using self-hosted hardware, only at a higher price point.
Re: Why did we take reddit down for 71 minutes?
#19Re: Why did we take reddit down for 71 minutes?
#20Earlier quoted context omitted.
Sounds like it's time to look at Redis...
Or Tokyo Cabinet / Tyrant, which is still a high performance key/value store but doesn't need to fit everything in RAM. Depends on how much they're storing.
If the data isn't "optional" (i.e. if there's a cache miss, you can't just go to a traditional database for a somewhat higher - but acceptable - cost), "memcached with persistence" approach isn't going to cut it. You now have a distributed system with state, which is a much more difficult problem.
That's why there are so many distributed storage systems: there's no "one fits all" solution that can handle every theoretical corner case (even Google hasn't solved that). In my (biased) view, the eventually consistent stores (e.g. Dynamo-inspired ones or even "Friendfeed/Facebook model" of sharded and replicated MySQL databases storing Blobs) do seem most reasonable for web-type problems.