Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Architecture Twitter Uses to Deal with 150M Active Users

highscalability.com

Post reply on HN
Print view View original

101–110 of 166 posts

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#101

Post by vidarh »

LewisJEllis wrote: post #17
papsosouid wrote: post #10

I really question the current trend of creating big, complex, fragile architectures to "be able to scale". These numbers are a great example of why, the entire thing could run on a single server, in a very straight forward setup. When you are creating a cluster for scalability, and it has less CPU, RAM and IO than a single server, what are you gaining? They are only doing 6k writes a second for crying out loud.

It's not as simple as 6k writes - it's 6k write requests. Most of these 'write requests' are tweets which must be written to the timelines of hundreds or thousands (or in some cases millions!) of followers. If you try to run it all on a single server, and Obama and Justin Bieber tweet at the same time, you suddenly have a backlog of 75 million writes to catch up on. Now imagine what happens if Bieber starts a convers…

Agreed that this is an issue, but it's a smaller one than you make out: Consider that there's likely a marked hockey stick going on here. Now keep the very most recent tweets of the very "worst" in terms of followers in a hot cache, and mix their most recent tweets into the timelines of their followers as needed until they're committed everywhere. Then collapse writes, and you reduce the "conversation" problem.

It's still not cheap to handle, but consider that fan-out on write is essentially an optimization. You don't need that optimization to be "pure" in that you don't need to depend on the writes being completed in a timely fashion - there are any numbers of tradeoffs you can make between read and write performance.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#102

Post by johnbellone »

papsosouid wrote: post #40
rythie wrote: post #37

Earlier quoted context omitted.

$1,000,000 machines are not that much faster, you won't get 1000x the performance or anywhere near. The memory and IO performance is going to be within a factor 2 or 4 of that high end machine. It might have 50x the cores, but most likely that's not the limiting factor anyway.

The whole point is you can get equal performance from a single server instead of a ton of little ones. The ton of little ones forced them to totally re-architect to work around the massive latency between servers. A single server would have allowed them to stick with a sane architecture, and saved them millions in development time and maintenance nightmares.

I'm sure you could get vastly better performance. But this is not about performance.

What happens when you need multiple datacenters? How about if you need to plan maintenance in a single datacenter? Therefore you need at least two servers in each datacenter, etc.

Let's say you decide to use smaller machines to serve the frontend but your backend machines are big iron. Are you going to perform all your computation and then push the data out to edge servers?

There's much more to this then loading up on memory and cores.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#103

Post by sk00byd00 »

xb95 wrote: post #35
papsosouid wrote: post #10

I really question the current trend of creating big, complex, fragile architectures to "be able to scale". These numbers are a great example of why, the entire thing could run on a single server, in a very straight forward setup. When you are creating a cluster for scalability, and it has less CPU, RAM and IO than a single server, what are you gaining? They are only doing 6k writes a second for crying out loud.

I think that in a lot of ways it's because of the roots of the organizations in question. Instead of saying "yeah, we have money, we can go buy this solution and trust in the vendor to support us" (an objectively viable solution in many situations) we instead say "we've got $100,000 in seed money and one engineer plus some friends she's convinced to help out pro bono, that won't buy us a sideways glance from Oracle".…

Your expensive vendors sit on security patches for weeks and my free ones shoot them out almost immediately. Why is that not a business concern?

I don't think it's zelotry, there are a lot of advantages to open source software you aren't going to get from Microsoft. If you are an edge case and you happen to stumble on that race condition bug are you going to have your engineers black box test and reverse engineer someone elses product illegally while they wait around for Microsoft support or have them look under the hood, patch the bug and move on?

What the big licenses fees get you is accountability, which is of course a huge thing, but what open source gives you is control.

I think if your business is software, then open source makes perfect business sense, especially if one of your assets is a team of competent engineers.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#104

Post by logic »

jeffasinger wrote: post #95
wdewind wrote: post #92

Earlier quoted context omitted.

I have not used nginx, and have never heard of it spoken about in such glowing terms. Can you maybe compare/contrast it with apache?

The biggest difference is that nginx uses an evented I/O, and handles many connections per thread/process, Apache can only handle one at a time. This allows it to to use far less memory per connection, and lets it perform reasonably under very high loads. It also has very low latency, even for small loads, and is relatively easy to install and configure on most platforms. nginx does an excellent job as a reverse prox…

FYI: http://httpd.apache.org/docs/current/mod/event.html

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#105

Post by StavrosK »

webwanderings wrote: post #4

> Twitter knows a lot about you from who you follow and what links you click on. No kidding. But we don't care as we live in the glass house of a celebrity culture. PS: downvote the quoted text if you must. My point is not obvious.

You realize it's public data, right? When I tweet something, I don't expect it to be private. HN knows a lot about me too.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#106

Post by apaprocki »

Since tweets are very much like financial ticks (fixed size, tiny), we can put the numbers in perspective. Let's compare Twitter vs say, OPRA (North American options), a single "exchange" feed.

Twitter: 300K QPS, firehose 22 MB/sec, 400 million tweets/day

OPRA: 12.73M MPS, firehose 3.36 GB/sec, 26.4 billion messages/day

http://opradata.com/specs/2012_2013_Traffic_Projections.pdf

edit: Also worth noting, the diff between OPRA 1/1/13 & 7/1/13 is ~256 MB/sec, 2.0 billion messages/day. So in just 6 months the existing firehose increased roughly 10x Twitter's entire output and roughly 5x'd the number of messages/day.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#107

Post by sumzup »

harryh wrote: post #84
mikemoka wrote: post #26

I am playing the armchair architect and my question will be probably wrong in infinite ways,but I might learn something, what is the reason why the service has to write a tweet on two million timelines, wouldn't it be cheaper if they let the client build the page on its own via restful apis?

That is actually a very interesting question and it turns out that whether it's better to fanout on write or on read depends on a few different things. There's a very widely read paper on the subject you might enjoy: http://research.yahoo.net/node/3203

The link doesn't seem to work (error: "Unable to connect to database server"); do you have an alternate link?

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#108

Post by squarecog »

apaprocki wrote: post #106

Since tweets are very much like financial ticks (fixed size, tiny), we can put the numbers in perspective. Let's compare Twitter vs say, OPRA (North American options), a single "exchange" feed. Twitter: 300K QPS, firehose 22 MB/sec, 400 million tweets/day OPRA: 12.73M MPS, firehose 3.36 GB/sec, 26.4 billion messages/day http://opradata.com/specs/2012_2013_Traffic_Projections.pdf edit: Also worth noting, the diff betw…

It's not the data coming in that makes things hard. It's the fanout of messages into their subscribers' timelines.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#109

Post by nahname »

nasalgoat wrote: post #94
herge wrote: post #78

Earlier quoted context omitted.

Memcache?

Redis is memcache only better.

What's better about Redis? Many caching strategies require the storage to auto expire, memcache does this automatically. Redis does not do this (I believe). Is it significantly faster?

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#110

Post by thezilch »

apaprocki wrote: post #106

Since tweets are very much like financial ticks (fixed size, tiny), we can put the numbers in perspective. Let's compare Twitter vs say, OPRA (North American options), a single "exchange" feed. Twitter: 300K QPS, firehose 22 MB/sec, 400 million tweets/day OPRA: 12.73M MPS, firehose 3.36 GB/sec, 26.4 billion messages/day http://opradata.com/specs/2012_2013_Traffic_Projections.pdf edit: Also worth noting, the diff betw…

Tweets are fanned out to more than a single feed and, in the "most important" cases, millions of feeds -- 31 million feeds for Lady Gaga. Your comparing 300K reads, which might not even include the sub-queries, to 12.7M writes. A single tweet from a single user would trump the writes; it's unclear whether it could be done in the same one-second window.

Twitter already does 30B Timeline deliveries a day, compared to the 26.4B of OPRA. Again, there is no telling what can be implied by a "Timeline delivery;" does it include pulling user and other secondary and tertiary objects? An HTML renderer? The doesn't say much for the capabilities and focuses on what they do.

There's also little comparison to be made on what powers OPRA. If Twitter can simply add nodes to their architecture, are they doing it wrong?

Post reply on HN