Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Architecture Twitter Uses to Deal with 150M Active Users

highscalability.com

Post reply on HN
Print view View original

141–150 of 166 posts

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#141

Post by sheldoan »

est wrote: post #111
nahname wrote: post #109

Earlier quoted context omitted.

What's better about Redis? Many caching strategies require the storage to auto expire, memcache does this automatically. Redis does not do this (I believe). Is it significantly faster?

A simple Google search defeats your belief http://redis.io/commands/set > Options > EX seconds -- Set the specified expire time, in seconds. http://redis.io/commands/expireat > EXPIREAT has the same effect and semantic as EXPIRE, but instead of specifying the number of seconds representing the TTL (time to live), it takes an absolute Unix timestamp (seconds since January 1, 1970)

In addition, the Redis config file allows you to specify an eviction policy when the maximum memory limit is reached:

  # volatile-lru -> remove the key with an expire set using an LRU algorithm
  # allkeys-lru -> remove any key accordingly to the LRU algorithm
  # volatile-random -> remove a random key with an expire set
  # allkeys-random -> remove a random key, any key
  # volatile-ttl -> remove the key with the nearest expire time (minor TTL)
  # noeviction -> don't expire at all, just return an error on write operations
Source: https://raw.github.com/antirez/redis/2.6/redis.conf

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#142

Post by sheldoan »

sumzup wrote: post #107
harryh wrote: post #84

Earlier quoted context omitted.

That is actually a very interesting question and it turns out that whether it's better to fanout on write or on read depends on a few different things. There's a very widely read paper on the subject you might enjoy: http://research.yahoo.net/node/3203

The link doesn't seem to work (error: "Unable to connect to database server"); do you have an alternate link?

Not a link to the full paper, but a summary is presented here: http://highscalability.com/blog/2012/1/17/paper-feeding-fren...

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#143

Post by sheldoan »

pesenti wrote: post #76

Does anybody know how that compares to Facebook? I believe they do not use write fanout but instead rely on the search federation model (the model claimed not to have worked for Twitter).

Facebook uses a fan-out-on-read approach

http://www.quora.com/Facebook-Engineering/Did-Facebook-devel...

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#144

Post by thezilch »

ak39 wrote: post #139
squarecog wrote: post #108

Earlier quoted context omitted.

It's not the data coming in that makes things hard. It's the fanout of messages into their subscribers' timelines.

I've read before that Twitter uses NoSQL (or non-relational databases) and that each tweet requires a copy to be pushed to each follower every time someone tweets (same for updates or deletes). I am of a very much traditional RDBMS mindset though I am willing to bend a bit :) ... but this idea of fanning out tweets (copying tweets) to each and every follower just sounds grossly inefficient. The amount of writes must…

The entire tweet is not copied; it is a per-user list [0] of 16B to 28B records [1] -- IDs to the tweet and tweet author and sometime adhoc bits and retweet ID.

[0] http://www.infoq.com/resource/presentations/Twitter-Timeline... [1] http://www.infoq.com/resource/presentations/Twitter-Timeline...

As for what you might be missing, a user will need the latest, for example, 50 tweets from only the 200 people they follow....

Do we pull in 200 * 50 latest-per-user [cached] tweets lists? Now we might have network IO issues; it's also slower and on every read; could possibly ignite some hot spots for certain read sites. We can solve for these things, but they are just different problems and not necessarily easier nor simpler.

Do we walk a [timestamp] range index with a composite of user IDs? We'll have to shard this index with 200M tweets/day, and there's not an easy way to keep the necessary joins or unions local or making sense of how many shards I should query asynchronously before discarding extras. I'll probably have to make several roundtrips with a supervising process, between synchronous fetches.

Then there is paging...

There is probably a sweet spot for not doing the fanout for those with large followings, which they touch on at the bottom of The Future. Ultimately, you have to pay the price somewhere on read or write; they've simply picked write, as it apparently is more predictable and manageable on their working set.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#145

Post by Myrmornis »

harryh wrote: post #84
mikemoka wrote: post #26

I am playing the armchair architect and my question will be probably wrong in infinite ways,but I might learn something, what is the reason why the service has to write a tweet on two million timelines, wouldn't it be cheaper if they let the client build the page on its own via restful apis?

That is actually a very interesting question and it turns out that whether it's better to fanout on write or on read depends on a few different things. There's a very widely read paper on the subject you might enjoy: http://research.yahoo.net/node/3203

Title? Link doesn't work.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#146

Post by antirez »

davidw wrote: post #52

> Your home timeline sits in a Redis cluster and has a maximum of 800 entries. Wow, that's pretty cool. Congrats to antirez - it must be a nice feeling knowing that your software powers such a big system!

Thank you David, I'm very happy indeed that many companies are using Redis to get work done, this is basically one of the biggest reasons why after 4 years I'm not giving up...

As you know I'm usually not the kind of guy focused to the task at hand for more than a limited timeframe, and then I've the temptation to switch to something else, but this time I'm finding the right motivations in the big user base.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#147

Post by Myrmornis »

webwanderings wrote: post #4

> Twitter knows a lot about you from who you follow and what links you click on. No kidding. But we don't care as we live in the glass house of a celebrity culture. PS: downvote the quoted text if you must. My point is not obvious.

The meaning of your metaphor certainly seems obvious; explicit even. Are you referring to a layer of more profound non-obviousness?

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#148

Post by rythie »

beagle3 wrote: post #134
mikemoka wrote: post #26

I am playing the armchair architect and my question will be probably wrong in infinite ways,but I might learn something, what is the reason why the service has to write a tweet on two million timelines, wouldn't it be cheaper if they let the client build the page on its own via restful apis?

Mostly because they chose the wrong backend technology (which they have been doing repeatedly since their early days). The right way to solve twitter would be to have 140-byte tweets sorted by a 64-bit key, with a few more attributes (all falls into 256-bytes neatly), shard them across servers and keep everything recent in memory. Logging into a server would fetch the list of following to the front end server, broadc…

My understanding is that Facebook does it that way. I think for Twitter it creates two problems though:

- It's hard to make that realtime, because you'd need to some sort of broadcast every second or at least every few seconds.

- It's hard to follow a lot of people (you could be getting a large number of replies), so there would need be a follow limit.

Facebook has a follow limit and people don't really expect Facebook to be realtime - the central bit is not really.

Also, there is no one right way to do these things, in my view.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#149

Post by tschellenbach »

Their setup is very similar to what we use at Fashiolista. (though of course we only have millions and not hundreds of millions of users). We've open sourced our approach and you can see an early example here: https://github.com/tschellenbach/Feedly/

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#150

Post by beagle3 »

rythie wrote: post #148
beagle3 wrote: post #134

Earlier quoted context omitted.

Mostly because they chose the wrong backend technology (which they have been doing repeatedly since their early days). The right way to solve twitter would be to have 140-byte tweets sorted by a 64-bit key, with a few more attributes (all falls into 256-bytes neatly), shard them across servers and keep everything recent in memory. Logging into a server would fetch the list of following to the front end server, broadc…

My understanding is that Facebook does it that way. I think for Twitter it creates two problems though: - It's hard to make that realtime, because you'd need to some sort of broadcast every second or at least every few seconds. - It's hard to follow a lot of people (you could be getting a large number of replies), so there would need be a follow limit. Facebook has a follow limit and people don't really expect Facebo…

> It's hard to make that realtime, because you'd need to some sort of broadcast every second or at least every few seconds.

Twitter isn't realtime either - they say they don't succeed to stay within the 5 seconds all the time, and when Gaga tweets it takes up to 5 minutes.

Furthermore, I was talking about broadcasting a request for updates on demand when needed. PGM/UDP can blast through hundreds of megabytes per second on gigabit connection. That's quite easy. 22MB/sec is nothing, even 100MB/sec is not much these days (though you might have to bond/team to make that reliable)

> It's hard to follow a lot of people (you could be getting a large number of replies), so there would need be a follow limit.

Not at all. Make the replies (at most) 5 tweets from each person you follow, with a "and there's more ..." flag in the reply, have the front end ask for more if it makes sense once the 50ms is done.

It's ok if the 300 people who follow one million people take 200ms instead of 50ms to get a reply. And if you want to make it quicker for them, have these ones (and only these ones) on a "push" rather than "pull" model. The vast majority of people follow less than 50 people, perhaps less than 20.

> Facebook has a follow limit and people don't really expect Facebook to be realtime - the central bit is not really.

People do expect facebook to be realtime, it mostly delivers (better than twitter), their limits are not hard (I know people who asked and got them lifted within a few minutes).

> Also, there is no one right way to do these things, in my view.

No, but there's a lot of wrong ways, and twitter keeps choosing among them.

Post reply on HN