Pervasive writes?
Why are Facebook, Digg and Twitter So Hard To Scale?
41–50 of 52 posts
Re: Why are Facebook, Digg and Twitter So Hard To Scale?
#42I've been wondering for a while why we don't see more often systems with a huge amount of RAM and asynchronous persistent updates, with a more, how should I put it, computer science-ish architecture inside. MVC works ok, but does it really mean we _have_ to use it always? Is it so hard to make system with a TeraByte of RAM? Why even use memcache? Why not make a clear decision that stuff like user statuses will never…
User statuses should never touch hard disk? What happens when your server restarts and everything you were storing in memory disappears?
Based on another article on highscalability, Twitter has one master MySQL server that is used for backups only. If a node crashes, it reloads that status from MySQL. That MySQL server handles 300 tweets per second @ roughly 2400 qps.
They now use Scala/JVM rather than rails for the backend, Rails for the frontend.
http://highscalability.com/scaling-twitter-making-twitter-10...
Re: Why are Facebook, Digg and Twitter So Hard To Scale?
#43I think it's laughable to put Twitter and Digg in the same category of scalability as Facebook. Maybe things have changed since the last time I visited Digg over 2 years ago, but the social networking aspects are not very significant. The vast majority of their hits are practically fully page cacheable. Twitter at least has an interesting scaling problem, but they don't have any features and they move at a glacial pa…
Facebook also "has about 30,000 servers supporting its operations, hosts 80 billion photos, and serves up more than 600,000 photos to its users every second."
(check out http://www.facebook.com/Engineering?v=wall )
Re: Why are Facebook, Digg and Twitter So Hard To Scale?
#44I've been wondering for a while why we don't see more often systems with a huge amount of RAM and asynchronous persistent updates, with a more, how should I put it, computer science-ish architecture inside. MVC works ok, but does it really mean we _have_ to use it always? Is it so hard to make system with a TeraByte of RAM? Why even use memcache? Why not make a clear decision that stuff like user statuses will never…
You can buy a machine with a terabyte of RAM. If your check doesn't bounce, IBM, HP, Sun, et al. will deliver it via forklift tomorrow. However, that box will cost much, much more than 16 boxes with 64GB each. Those 16 boxes will also have more CPU, network cards, and aggregate bandwidth to RAM than the terabox. The big honkin' box full of RAM is in most respects an unbalanced system; you can do a lot more with a lot less money if it's acceptable to give up on the single-system image programming model.
Re: Why are Facebook, Digg and Twitter So Hard To Scale?
#45Because those who ignore the pub-sub research IBM did in the 80s are doomed to re-implement it badly.
I'll preface this with the fact that I've not read those papers (but today was thinking through the concept of write hooks in information graphs), but I'd assume from the common formation of the pattern that it's only really set up for one-dimensional publishes. The problem in large scale information networks, more in the Facebook way than the Twitter way, is that you you potentially trigger a cascading effect in inf…
In some models, sure, but is that true in Facebook's? Does anything go beyond one degree?
I can't think of anything, really. If X and Y becomes friends, X's friends and Y's friends see it in their feed ("inbox"), but nobody else. Same goes for tagging someone in a photo, etc.
Re: Why are Facebook, Digg and Twitter So Hard To Scale?
#46I think it's laughable to put Twitter and Digg in the same category of scalability as Facebook. Maybe things have changed since the last time I visited Digg over 2 years ago, but the social networking aspects are not very significant. The vast majority of their hits are practically fully page cacheable. Twitter at least has an interesting scaling problem, but they don't have any features and they move at a glacial pa…
We (FB) actually push new code every day.
Re: Why are Facebook, Digg and Twitter So Hard To Scale?
#47Earlier quoted context omitted.
I'll preface this with the fact that I've not read those papers (but today was thinking through the concept of write hooks in information graphs), but I'd assume from the common formation of the pattern that it's only really set up for one-dimensional publishes. The problem in large scale information networks, more in the Facebook way than the Twitter way, is that you you potentially trigger a cascading effect in inf…
In some models, sure, but is that true in Facebook's? Does anything go beyond one degree? I can't think of anything, really. If X and Y becomes friends, X's friends and Y's friends see it in their feed ("inbox"), but nobody else. Same goes for tagging someone in a photo, etc.
Ad targeting, what's hot in your network, friend recommenders, etc. are all essentially ranking problems. They're moving more in that direction, definitely.
Re: Why are Facebook, Digg and Twitter So Hard To Scale?
#48I think it's laughable to put Twitter and Digg in the same category of scalability as Facebook. Maybe things have changed since the last time I visited Digg over 2 years ago, but the social networking aspects are not very significant. The vast majority of their hits are practically fully page cacheable. Twitter at least has an interesting scaling problem, but they don't have any features and they move at a glacial pa…
Facebook also "has about 30,000 servers supporting its operations, hosts 80 billion photos, and serves up more than 600,000 photos to its users every second." (check out http://www.facebook.com/Engineering?v=wall )
Facebook handles 30K machines, 300M active users, 20B photos, & 25TB per day of logging data. http://bit.ly/EA3oY
Re: Why are Facebook, Digg and Twitter So Hard To Scale?
#49Earlier quoted context omitted.
FB should be more difficult as well because they aren't just showing streaming information. They're making (supposedly) intelligent decisions on what information to show you. Twitter is just showing you a "dumb" stream.
At the same time, facebook can simply drop messages on the floor if they're bottlenecked, so they have less strict scaling requirements.
Twitter has guaranteed delivery? Are you sure?
The killer app of Facebook - poking - does need to be reliable ;-)
Re: Why are Facebook, Digg and Twitter So Hard To Scale?
#50Earlier quoted context omitted.
A blog post would be great. Don't you find it odd though, that the engineers at Digg, Twitter, and Facebook think it is a very difficult problem?
When you grow from zero to 100million users you get to watch as each piece breaks under growing load. Keeping a system running as it just keeps growing is hard. However, Facebook rolled out a messaging system with little problem. I think the problem is guessing and simulating the load before people start messing with it. While not wasting millions building for load that never shows up.
But they didn't. They rolled out the messaging code to the users with no interface on it at first (e.g. presence only, visible only internally). That gave them excellent information on how many people might be connected, how many of their friends were likely to be connected, and so on. By the time they were ready to deploy the interface to it, they had a good idea of how much power the needed under it.