I really like 'behind the scenes' stories like this from big sites like Reddit, Facebook, Heroku etc. I work on a smaller scale of a few EC2 at a time, but I really enjoy the scaling info.
Why did we take reddit down for 71 minutes?
31–40 of 47 posts
Re: Why did we take reddit down for 71 minutes?
#32"Memcachedb also has another feature that blocks all reads while it writes to the disk." Seriously? Wow.
Sounds like it's time to look at Redis...
Well, no; if they're using memcachedb presumably their data doesn't fit in memory on a single machine.
Cassandra would be a better choice.
/cassandra committer, but it would :)
Re: Why did we take reddit down for 71 minutes?
#33One major reason not to hop on the cloud bandwagon just yet is issues like these. The more layers underneath that are not under your control the more layers you'll have to add to remedy that. Systems with excessive complexity are hard to debug, especially when it comes to analyzing performance issues. Given complete control of the hardware from the ground up it can already be quite hard to accurately pinpoint a bottl…
Reddit went on the cloud bandwagon not because they thought it was superior to managed servers, but because Conde Nast's IT department sucked. With the ongoing growth, they were having trouble procuring the additional servers as needed, and moving Amazon solved that problem.
Can you give us any reference to the "Condé Nast's IT department's fault" theory? From the best of my knowledge reddit infrastructure was always maintained by reddit stuff and Condé Nast had no influence on that.
The current reddit stuff did an "Ask me Anything" thread on reddit when the founders left and they said the reason for the move to AWS was purely because of price/scaling issues and some part of reddit was already using AWS even before Condé Nast bought them.
Re: Why did we take reddit down for 71 minutes?
#34Earlier quoted context omitted.
This isn't an EC2 issue, this is a SAN issue. Wether it is EBS or an NFS drive, meh. This is an architecting issue and while it is a result of the underlying hardware, the underlying hardware is not the constraint.
agreed it's largely an architecture issue, however poor EBS performance is contributing factor and he seems to go out of his way to say that it's not...
He went out of his way because most reddit user keep blaming AWS for the recent issues, as jedberg (reddit IT guy) recently mentioned the problem is not with AWS scaling but with reddit software scaling.
Re: Why did we take reddit down for 71 minutes?
#35Earlier quoted context omitted.
Sounds like it's time to look at Redis...
Well, no; if they're using memcachedb presumably their data doesn't fit in memory on a single machine. Cassandra would be a better choice. /cassandra committer, but it would :)
We actually looked at Cassandra and found it to be slower than memcachedb. However, we readily admit that we probably configured it wrong.
Re: Why did we take reddit down for 71 minutes?
#36Does anyone know if reddit is doing anything special to create the RAID? Or is it just mdadm?
Just mdadm.
Re: Why did we take reddit down for 71 minutes?
#37One major reason not to hop on the cloud bandwagon just yet is issues like these. The more layers underneath that are not under your control the more layers you'll have to add to remedy that. Systems with excessive complexity are hard to debug, especially when it comes to analyzing performance issues. Given complete control of the hardware from the ground up it can already be quite hard to accurately pinpoint a bottl…
Actually, we did work with Amazon. The RAID was their suggestion.
Re: Why did we take reddit down for 71 minutes?
#38One major reason not to hop on the cloud bandwagon just yet is issues like these. The more layers underneath that are not under your control the more layers you'll have to add to remedy that. Systems with excessive complexity are hard to debug, especially when it comes to analyzing performance issues. Given complete control of the hardware from the ground up it can already be quite hard to accurately pinpoint a bottl…
Reddit went on the cloud bandwagon not because they thought it was superior to managed servers, but because Conde Nast's IT department sucked. With the ongoing growth, they were having trouble procuring the additional servers as needed, and moving Amazon solved that problem.
That's not entirely true. Yes, it is true that getting servers was hard, and that was definitely a factor.
But the bigger factor for me was that I was tired of having to build, image and rack all those servers. I liked the flexibility of EC2, and also not having to waste resources ordering a full rack's worth of hardware every time.
Cost was also an issue. Datacenter space in SF is expensive, but it had to be in SF, because that is where I was. EC2 proved to be much cheaper than physical servers.
I also like the fact that I don't have to run to the datacenter anymore when there is an issue. I just file a ticket with Amazon.
Re: Why did we take reddit down for 71 minutes?
#39Earlier quoted context omitted.
This isn't an EC2 issue, this is a SAN issue. Wether it is EBS or an NFS drive, meh. This is an architecting issue and while it is a result of the underlying hardware, the underlying hardware is not the constraint.
agreed it's largely an architecture issue, however poor EBS performance is contributing factor and he seems to go out of his way to say that it's not...
Blaming poor EBS performance would be like blaming Intel for their 4GhZ processor not being able to do your protein folding in 5 seconds.
It is simply a known limitation that has to be worked around.
Even if we owned the servers, we would have the same limitation -- eventually you just can't get the performance out of a single disk.
Re: Why did we take reddit down for 71 minutes?
#40Earlier quoted context omitted.
Reddit went on the cloud bandwagon not because they thought it was superior to managed servers, but because Conde Nast's IT department sucked. With the ongoing growth, they were having trouble procuring the additional servers as needed, and moving Amazon solved that problem.
That's not entirely true. Yes, it is true that getting servers was hard, and that was definitely a factor. But the bigger factor for me was that I was tired of having to build, image and rack all those servers. I liked the flexibility of EC2, and also not having to waste resources ordering a full rack's worth of hardware every time. Cost was also an issue. Datacenter space in SF is expensive, but it had to be in SF,…
OK, thanks for the clarification. I'm trying to find the post I read that gave me that idea but can't find it. Was it in in your AMA?