Live data from Hacker News

Why are Facebook, Digg and Twitter So Hard To Scale?

highscalability.com

21–30 of 52 posts

Re: Why are Facebook, Digg and Twitter So Hard To Scale?

#21
post #13

Because those who ignore the pub-sub research IBM did in the 80s are doomed to re-implement it badly.

I'll preface this with the fact that I've not read those papers (but today was thinking through the concept of write hooks in information graphs), but I'd assume from the common formation of the pattern that it's only really set up for one-dimensional publishes.

The problem in large scale information networks, more in the Facebook way than the Twitter way, is that you you potentially trigger a cascading effect in information updates if you go pub-sub. Specifically, applications that do interesting things with social graphs have to go beyond basically doing message passing.

I naturally look at things from a recommendations angle, but if you've got a new edge that enters the graph that may affect other edges that are connected to the end points. Those may in turn affect the edges that are connected to those nodes and so on. You want to avoid something that effectively becomes a breadth first traversal of the graph doing updates since that's well, slow, to put it mildly.

This is why large-scale graph algorithms like PageRank work on constantly regenerating static matrices rather than doing regeneration of the ranks dynamically, but that naturally is problematic when you're working on data sets where the most recent data is the most important and is being generated at very high rates.

Re: Why are Facebook, Digg and Twitter So Hard To Scale?

#22
post #9

I think it's laughable to put Twitter and Digg in the same category of scalability as Facebook. Maybe things have changed since the last time I visited Digg over 2 years ago, but the social networking aspects are not very significant. The vast majority of their hits are practically fully page cacheable. Twitter at least has an interesting scaling problem, but they don't have any features and they move at a glacial pa…

What makes Twitter hard is that it is a semi-real-time communications medium that is no longer particularly web-based - a good portion of Tweets come from phones.

But you're right with Digg, I don't see it having any excuses

Re: Why are Facebook, Digg and Twitter So Hard To Scale?

#23
post #9

I think it's laughable to put Twitter and Digg in the same category of scalability as Facebook. Maybe things have changed since the last time I visited Digg over 2 years ago, but the social networking aspects are not very significant. The vast majority of their hits are practically fully page cacheable. Twitter at least has an interesting scaling problem, but they don't have any features and they move at a glacial pa…

[deleted]

Re: Why are Facebook, Digg and Twitter So Hard To Scale?

#24
post #9

I think it's laughable to put Twitter and Digg in the same category of scalability as Facebook. Maybe things have changed since the last time I visited Digg over 2 years ago, but the social networking aspects are not very significant. The vast majority of their hits are practically fully page cacheable. Twitter at least has an interesting scaling problem, but they don't have any features and they move at a glacial pa…

There are facebook pages with millions of fans. If you fan a page, you'll "subscribe" its updates too. It is similar to asymmetric following.

The internal representation is quite elegant.

Re: Why are Facebook, Digg and Twitter So Hard To Scale?

#25
I've been wondering for a while why we don't see more often systems with a huge amount of RAM and asynchronous persistent updates, with a more, how should I put it, computer science-ish architecture inside. MVC works ok, but does it really mean we _have_ to use it always? Is it so hard to make system with a TeraByte of RAM? Why even use memcache? Why not make a clear decision that stuff like user statuses will never even see the inside of a hard disk and make use of the simplification it brings? Is there a clear reason, or simply it's not fashionable?

Re: Why are Facebook, Digg and Twitter So Hard To Scale?

#26
post #10
post #3

Earlier quoted context omitted.

Sure it's obvious until you have hundreds of millions of nodes that each link to somewhere between 100 and 2,000,000 other nodes that need to be updated when any given node updates (assume around 5,000 nodes are updating every second with a power law kind of distribution). If you think that's obvious, I think you're significantly beyond all of the people with all of the "hello world" twitter clones out there.

It all comes down how well you can slice the process. There is no cheep over the counter solution to these problems but a little custom code can go a long way. 5k node updates per second might sound like a problem, but one core of one machine can easily keep up with that so you can have several copies and several views of the whole network graph. Public vs. private messages can be handled separately and then joined b…

Twitter was forced to morph an architecture built to solve a different problem into a working solution.

BINGO! Twitter's initial implementation was a "my first blog" in Rails -- when you're starting from an impedance mismatch that massive, and you have to rearchitect it live, jesus.

Re: Why are Facebook, Digg and Twitter So Hard To Scale?

#27
post #9

I think it's laughable to put Twitter and Digg in the same category of scalability as Facebook. Maybe things have changed since the last time I visited Digg over 2 years ago, but the social networking aspects are not very significant. The vast majority of their hits are practically fully page cacheable. Twitter at least has an interesting scaling problem, but they don't have any features and they move at a glacial pa…

FB should be more difficult as well because they aren't just showing streaming information. They're making (supposedly) intelligent decisions on what information to show you. Twitter is just showing you a "dumb" stream.

Re: Why are Facebook, Digg and Twitter So Hard To Scale?

#28
post #4

I've wondered for a while about some hybrid push/pull distribution scheme. I have 100 followers, just push. Oprah has 1M, 1% of which are currently active, pull. Sounds complicated, but feasible.

But is also has to do with how frequently the information is changed/accessed.

e.g. If Oprah only publishes 3 tweets a day, but her one million followers each check 100 times a day (just to be on the bleeding edge of gossip), it's much less effort to push the change.

On the flip side, if you post status changes several times a day but your followers rarely check (daily/weekly), pull may make more sense.

Although, I suppose that a push inherently requires an update/write, while a pull is generally a read. Seems like this might need to be taken into consideration as well.

Re: Why are Facebook, Digg and Twitter So Hard To Scale?

#29
post #9

I think it's laughable to put Twitter and Digg in the same category of scalability as Facebook. Maybe things have changed since the last time I visited Digg over 2 years ago, but the social networking aspects are not very significant. The vast majority of their hits are practically fully page cacheable. Twitter at least has an interesting scaling problem, but they don't have any features and they move at a glacial pa…

Just because Twitter and Digg don't operate at the same scale as Facebook doesn't mean they don't face similar issues.

I actually think having a bunch of large sites tackling similar issues will present a whole bunch of neat new technologies for addressing these sorts of issues in the near future.

Re: Why are Facebook, Digg and Twitter So Hard To Scale?

#30
post #27
post #9

I think it's laughable to put Twitter and Digg in the same category of scalability as Facebook. Maybe things have changed since the last time I visited Digg over 2 years ago, but the social networking aspects are not very significant. The vast majority of their hits are practically fully page cacheable. Twitter at least has an interesting scaling problem, but they don't have any features and they move at a glacial pa…

FB should be more difficult as well because they aren't just showing streaming information. They're making (supposedly) intelligent decisions on what information to show you. Twitter is just showing you a "dumb" stream.

At the same time, facebook can simply drop messages on the floor if they're bottlenecked, so they have less strict scaling requirements.
Post reply on HN