Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Architecture Twitter Uses to Deal with 150M Active Users

highscalability.com

Post reply on HN
Print view View original

131–140 of 166 posts

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#131

Post by coldtea »

wdewind wrote: post #92
nasalgoat wrote: post #65

Earlier quoted context omitted.

Redis is probably the most useful tool powering the internet after nginx. It really is an amazing piece of engineering.

I have not used nginx, and have never heard of it spoken about in such glowing terms. Can you maybe compare/contrast it with apache?

>I have not used nginx, and have never heard of it spoken about in such glowing terms.

Seriously? There are lots of glowing blog posts, articles etc about Nginx all the time, including on HN. Just last week or so, it was reported that it powers the majority of the top-1000 biggest sites.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#132

Post by bitbckt »

joshuaellinger wrote: post #127

The surprise for me is that the core component is Redis. My first guess would have been custom C code. Yeah, you have to do everything yourself. Yeah, it would be hard to write. But you'd control every little bit of it. Obviously, I must not fully understand the problem and what Redis buys. Sam Puralla (if you are reading) -- do you know why didn't Twitter go with a full custom system at its heart? Josh

I'm probably a better person to answer this than Sam - I'm a former lead on this project - so I'll take a swing:

We chose Redis because it gave us the specific, incremental improvements over our existing memcached-based system that we required, without requiring us to write (yet another) component. There was enough to do, and this choice has turned out to be good enough, I think.

As the project progressed though, we treated Redis much like we did own it. We altered the wire protocol, changed the eviction strategy, and reduced protocol parsing overhead, for example. Much of that work has long since made it upstream.

[edit] grammar

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#133

Post by jacques_chester »

squarecog wrote: post #130
jacques_chester wrote: post #121

Figuratively, Twitter have switched from doing a design rooted in Databases 101 (OLTP) to a design more rooted in Databases 102 (OLAP). That is, they moved processing from query time to write time. And that's a perfectly legitimate strategy; it's the basis of data warehousing. OLTP is about write-time speed. It's great for stuff like credit card transactions, where you only really care about the tally at the end of t…

That's an interesting thought, but rather than focusing on write vs read balance, I would claim that OLAP and OLTP are most distinguished by the nature of queries they need to support. OLAP is characterized by fairly low-volume aggregation queries that touch very large volumes of data ("what fraction of tweets in English come from non-english speaking countries?"). Sure, the ingest is large, but it tends to be batch…

Thanks, I hadn't mentally partitioned OLAP/OLTP that way. I still think that the read/write distinction more correctly forms the dividing line.

That's because the classical OLTP approach is to use relational databases. Relational databases do well in writes because in a properly normalised DB there is one, and only one, place for each datum to go. There is no fan out and potentially no coordination. A place for everything and everything in its place. That reduces the amount of write traffic required to support complex models of the problem domain by a lot.

But of course the relational view of a problem domain doesn't really look like the humanistic view of a problem domain. And building the humanistic view of the problem domain usually means denormalising and breaking the things that made OLTP useful. Enter ETL pipes and OLAP systems.

From there it's very humanistic and it requires no understanding of the relational model. Tools can easily turn the dimensional tables into dropdown filter lists that look like a spreadsheet -- or even run inside a spreadsheet.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#134

Post by beagle3 »

mikemoka wrote: post #26

I am playing the armchair architect and my question will be probably wrong in infinite ways,but I might learn something, what is the reason why the service has to write a tweet on two million timelines, wouldn't it be cheaper if they let the client build the page on its own via restful apis?

Mostly because they chose the wrong backend technology (which they have been doing repeatedly since their early days).

The right way to solve twitter would be to have 140-byte tweets sorted by a 64-bit key, with a few more attributes (all falls into 256-bytes neatly), shard them across servers and keep everything recent in memory.

Logging into a server would fetch the list of following to the front end server, broadcast the request to all tweet servers, wait 50ms or so for responses, merge, sort and HTML format them.

The front end servers would not need any memory or disk (could be an army of $500 servers behind a load balance, or a few beefy ones). The backend servers would have to have some beefy CPU and memory, but still ultra commodity (256 bytes/tween means 1GB=4M tweets, so one 64GB server=256M tweets). Shard for latency, redundancy, etc. Also, special case the Gagas/Kutchers of this world by giving them their own server, and/or have them broadcast to and cache their tweets in the front end servers (Spend 256MB memory on tweet cache in the front end servers, and you get 1M cached tweets - which would cover all of the popular people and then some).

Network broadcast was invented for a reason.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#135

Post by granitepail »

apaprocki wrote: post #124
jacques_chester wrote: post #118

Earlier quoted context omitted.

Securities feeds are, I suspect, more write-dominant and have a much lower fanout rate. They also don't have the outlier-retweeting-outlier or outlier-tweeting-@-outlier problems.

Yes, much more write-dominant. Apps which are built on top of the feeds can create issues, though. E.g., An obscure ticker pasted into a chat room with 500 people instantly starts monitoring in all their windows -- not just the static price (equiv to a RT), but the live feed. That would be as if A RT'd B and suddenly all of A's followers added B to their timeline. The degree to which that happens depends on how many…

I have no idea why you're getting so much pushback. Products like those offered by Bloomberg ingest and distribute massive amounts of data in real time, as well. The comparison is completely suitable.

While their products for investment managers are mostly run off of persistent databases, the trader terminals rely on a high-volume, nebulous fan out. For many traders, a five second latency is unacceptable.

Incredibly interesting talk and a good write up. Twitter continues to impress! I was surprised to see Redis playing such a critical role, too.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#136

Post by beagle3 »

logic wrote: post #57
mikemoka wrote: post #26

I am playing the armchair architect and my question will be probably wrong in infinite ways,but I might learn something, what is the reason why the service has to write a tweet on two million timelines, wouldn't it be cheaper if they let the client build the page on its own via restful apis?

In addition to what others have mentioned, consider passive endpoints, such as SMS and push notifications.

But they don't all have to fit within the same framework - indeed, don't fit into the same framework.

e.g., if you can't send the SMS or push right now, you have to do a retry 5 minutes later. If you can't update a followers queue (if that's what you do), you have to display a fail whale....

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#137

Post by _pmf_ »

joshuaellinger wrote: post #127

The surprise for me is that the core component is Redis. My first guess would have been custom C code. Yeah, you have to do everything yourself. Yeah, it would be hard to write. But you'd control every little bit of it. Obviously, I must not fully understand the problem and what Redis buys. Sam Puralla (if you are reading) -- do you know why didn't Twitter go with a full custom system at its heart? Josh

> The surprise for me is that the core component is Redis.

> My first guess would have been custom C code.

I'm pretty sure they use a heavily tailored Redis, so both is true.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#138

Post by jacques_chester »

granitepail wrote: post #135
apaprocki wrote: post #124

Earlier quoted context omitted.

Yes, much more write-dominant. Apps which are built on top of the feeds can create issues, though. E.g., An obscure ticker pasted into a chat room with 500 people instantly starts monitoring in all their windows -- not just the static price (equiv to a RT), but the live feed. That would be as if A RT'd B and suddenly all of A's followers added B to their timeline. The degree to which that happens depends on how many…

I have no idea why you're getting so much pushback. Products like those offered by Bloomberg ingest and distribute massive amounts of data in real time, as well. The comparison is completely suitable. While their products for investment managers are mostly run off of persistent databases, the trader terminals rely on a high-volume, nebulous fan out. For many traders, a five second latency is unacceptable. Incredibly…

Sometimes the pushback is right, sometimes it's wrong. He's been very good at explaining the different and heavy requirements of financial data feeds.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#139

Post by ak39 »

squarecog wrote: post #108
apaprocki wrote: post #106

Since tweets are very much like financial ticks (fixed size, tiny), we can put the numbers in perspective. Let's compare Twitter vs say, OPRA (North American options), a single "exchange" feed. Twitter: 300K QPS, firehose 22 MB/sec, 400 million tweets/day OPRA: 12.73M MPS, firehose 3.36 GB/sec, 26.4 billion messages/day http://opradata.com/specs/2012_2013_Traffic_Projections.pdf edit: Also worth noting, the diff betw…

It's not the data coming in that makes things hard. It's the fanout of messages into their subscribers' timelines.

I've read before that Twitter uses NoSQL (or non-relational databases) and that each tweet requires a copy to be pushed to each follower every time someone tweets (same for updates or deletes). I am of a very much traditional RDBMS mindset though I am willing to bend a bit :) ... but this idea of fanning out tweets (copying tweets) to each and every follower just sounds grossly inefficient. The amount of writes must be exponential.

The way I see it, here are the WRITES they'll ever need:

1:User -> n:Users (Followers) (Temporal links with effective dates for follow history)

1:User -> n:Tweets (Temporal with tweet datetime) - this includes Retweets.

The rest are just READS (pulls instead of fanning out the same data).

What am I missing here folks? :-) Educate me please.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#140

Post by Myrmornis »

ak39 wrote: post #139
squarecog wrote: post #108

Earlier quoted context omitted.

It's not the data coming in that makes things hard. It's the fanout of messages into their subscribers' timelines.

I've read before that Twitter uses NoSQL (or non-relational databases) and that each tweet requires a copy to be pushed to each follower every time someone tweets (same for updates or deletes). I am of a very much traditional RDBMS mindset though I am willing to bend a bit :) ... but this idea of fanning out tweets (copying tweets) to each and every follower just sounds grossly inefficient. The amount of writes must…

I think what you're missing is simply that they have explicitly decided not to compute the home timeline on request; they've decided that they need it to be precomputed and constantly updated.
Post reply on HN