The Architecture Twitter Uses to Deal with 150M Active Users • Hacker News Board

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#111

Post by est » Tue, Jul 09, 2013, 2:38 AM UTC

nahname wrote: post #109

nasalgoat wrote: post #94

Earlier quoted context omitted.

Redis is memcache only better.

What's better about Redis? Many caching strategies require the storage to auto expire, memcache does this automatically. Redis does not do this (I believe). Is it significantly faster?

A simple Google search defeats your belief

http://redis.io/commands/set

> Options

> EX seconds -- Set the specified expire time, in seconds.

http://redis.io/commands/expireat

> EXPIREAT has the same effect and semantic as EXPIRE, but instead of specifying the number of seconds representing the TTL (time to live), it takes an absolute Unix timestamp (seconds since January 1, 1970)

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#112

Post by spiffytech » Tue, Jul 09, 2013, 2:39 AM UTC

nasalgoat wrote: post #94

herge wrote: post #78

Earlier quoted context omitted.

Memcache?

Redis is memcache only better.

This[0] StackOverflow answer nicely lays this fact out.

[0]: http://stackoverflow.com/a/11257333/191438

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#113

Post by binarycrusader » Tue, Jul 09, 2013, 2:39 AM UTC

ariwilson wrote: post #74

phwamgflw wrote: post #59

Earlier quoted context omitted.

Not posting this to support his argument, but for the record some of the high end unix hardware available (for a price, no idea what these cost): 32TB RAM 1024 Cores (64 x 16 core), 928 x PCI Express I/O slots: http://www.oracle.com/us/products/servers-storage/servers/sp... 16TB RAM 256 cores (probably multiple threads per core), 640 x PCIe I/O adapters: http://www-03.ibm.com/systems/power/hardware/795/specs.html 4TB…

And exactly to his point - you still can't treat these as one uniform huge memory / computational space for your application (these machines seem designed for virtualization rather than one huge application). You run into the same distributed computing issues you would with your own hardware, just with a 5/10x larger initial investment and without a huge amount of pricing control / flexibility in terms of adding capa…

Actually you can treat these as one uniform huge memory / computational space for your application. They're not meant only for virtualisation. In particular, the Oracle Database is a perfect fit for a system with thousands of cores and terabytes of memory.

It's true that for some use cases, you'd be better off carving it up using some form of virtualisation, but it isn't a requirement to reap the benefits of a massive system.

Both the Solaris scheduler and virtual memory system are designed for the kind of scalability needed when working with thousands of cores and terabytes of memory.

You also don't run into the same distributed system issues when you use the system that way.

You also do actually have a fair amount of flexibility in dealing with failures as they arise. Solaris has extensive support for DR (Dynamic Reconfiguration). In short, CPUs can be hot-swapped if needed, and memory can also be removed or added dynamically.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#114

Post by apaprocki » Tue, Jul 09, 2013, 2:46 AM UTC

thezilch wrote: post #110

apaprocki wrote: post #106

Since tweets are very much like financial ticks (fixed size, tiny), we can put the numbers in perspective. Let's compare Twitter vs say, OPRA (North American options), a single "exchange" feed. Twitter: 300K QPS, firehose 22 MB/sec, 400 million tweets/day OPRA: 12.73M MPS, firehose 3.36 GB/sec, 26.4 billion messages/day http://opradata.com/specs/2012_2013_Traffic_Projections.pdf edit: Also worth noting, the diff betw…

Tweets are fanned out to more than a single feed and, in the "most important" cases, millions of feeds -- 31 million feeds for Lady Gaga. Your comparing 300K reads, which might not even include the sub-queries, to 12.7M writes. A single tweet from a single user would trump the writes; it's unclear whether it could be done in the same one-second window. Twitter already does 30B Timeline deliveries a day, compared to t…

Securities work a similar same way -- When you're looking at your portfolio you're only directly monitoring the individual fields for the securities in your portfolio, not the entire firehose. The OPRA feed is an equivalent to the raw Twitter firehose, only much fatter. Once you integrate it into the last-mile display to the users, you're doing all the same things. This is typically done with multicast topic-style subscriptions by ticker / type of data. It makes sense that if Lady Gaga writes a tweet you only want it to be "1 timeline delivery" (multicast) as opposed to "31 million timeline deliveries", which is making your infrastructure do a lot more work. Granted, you're kind of limited by what the browser can do in this regard, so you're kind of stuck with the socket model.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#115

Post by dchichkov » Tue, Jul 09, 2013, 2:52 AM UTC

apaprocki wrote: post #106

Since tweets are very much like financial ticks (fixed size, tiny), we can put the numbers in perspective. Let's compare Twitter vs say, OPRA (North American options), a single "exchange" feed. Twitter: 300K QPS, firehose 22 MB/sec, 400 million tweets/day OPRA: 12.73M MPS, firehose 3.36 GB/sec, 26.4 billion messages/day http://opradata.com/specs/2012_2013_Traffic_Projections.pdf edit: Also worth noting, the diff betw…

300K QPS is pretty impressive. If you'd want to sustain that rate on a single modern machine you'll have to fit all your processing into about thirty LLC misses per request.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#116

Post by apaprocki » Tue, Jul 09, 2013, 2:55 AM UTC

dchichkov wrote: post #115

apaprocki wrote: post #106

Since tweets are very much like financial ticks (fixed size, tiny), we can put the numbers in perspective. Let's compare Twitter vs say, OPRA (North American options), a single "exchange" feed. Twitter: 300K QPS, firehose 22 MB/sec, 400 million tweets/day OPRA: 12.73M MPS, firehose 3.36 GB/sec, 26.4 billion messages/day http://opradata.com/specs/2012_2013_Traffic_Projections.pdf edit: Also worth noting, the diff betw…

300K QPS is pretty impressive. If you'd want to sustain that rate on a single modern machine you'll have to fit all your processing into about thirty LLC misses per request.

I admit the QPS/MPS thing doesn't really fit. I mainly wanted to illustrate the data rates of the raw feeds. Twitter has one of the largest raw feeds of the web world, but it's useful to know how that compares to other non-web feeds.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#117

Post by jebblue » Tue, Jul 09, 2013, 3:20 AM UTC

smandou wrote: post #69

nasalgoat wrote: post #65

Earlier quoted context omitted.

Redis is probably the most useful tool powering the internet after nginx. It really is an amazing piece of engineering.

You forgot the JVM.

Yup that, Eclipse, vi, Apache, MySQL and Oracle are my bread and butter. Never used nginx or Redis.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#118

Post by jacques_chester » Tue, Jul 09, 2013, 3:22 AM UTC

apaprocki wrote: post #114

thezilch wrote: post #110

Earlier quoted context omitted.

Tweets are fanned out to more than a single feed and, in the "most important" cases, millions of feeds -- 31 million feeds for Lady Gaga. Your comparing 300K reads, which might not even include the sub-queries, to 12.7M writes. A single tweet from a single user would trump the writes; it's unclear whether it could be done in the same one-second window. Twitter already does 30B Timeline deliveries a day, compared to t…

Securities work a similar same way -- When you're looking at your portfolio you're only directly monitoring the individual fields for the securities in your portfolio, not the entire firehose. The OPRA feed is an equivalent to the raw Twitter firehose, only much fatter. Once you integrate it into the last-mile display to the users, you're doing all the same things. This is typically done with multicast topic-style su…

Securities feeds are, I suspect, more write-dominant and have a much lower fanout rate. They also don't have the outlier-retweeting-outlier or outlier-tweeting-@-outlier problems.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#119

Post by jacques_chester » Tue, Jul 09, 2013, 3:27 AM UTC

davidgerard wrote: post #77

papsosouid wrote: post #10

I really question the current trend of creating big, complex, fragile architectures to "be able to scale". These numbers are a great example of why, the entire thing could run on a single server, in a very straight forward setup. When you are creating a cluster for scalability, and it has less CPU, RAM and IO than a single server, what are you gaining? They are only doing 6k writes a second for crying out loud.

I do consider it a pathology when tiny services, or tiny apps in a corporate structure, act like they have the problems of Google. You are not Google. You do not have Google's problems. You do not have scaling issues. For you, N is small and will stay small . Stop giving me this delusional resume-padding garbage to implement. For you, here, it is delusion and lies.

The point here is that Twitter really is one of the cases where vertical scaling is not, on the balance of non-functional requirements to engineering overhead, the right decision. They really do need to pay the complexity piper.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#120

Post by thezilch » Tue, Jul 09, 2013, 3:30 AM UTC

apaprocki wrote: post #114

thezilch wrote: post #110

Earlier quoted context omitted.

Tweets are fanned out to more than a single feed and, in the "most important" cases, millions of feeds -- 31 million feeds for Lady Gaga. Your comparing 300K reads, which might not even include the sub-queries, to 12.7M writes. A single tweet from a single user would trump the writes; it's unclear whether it could be done in the same one-second window. Twitter already does 30B Timeline deliveries a day, compared to t…

Securities work a similar same way -- When you're looking at your portfolio you're only directly monitoring the individual fields for the securities in your portfolio, not the entire firehose. The OPRA feed is an equivalent to the raw Twitter firehose, only much fatter. Once you integrate it into the last-mile display to the users, you're doing all the same things. This is typically done with multicast topic-style su…

I think we're conflating topics. While the Firehose can be done with some combination of Pub-Sub/Topic/Sub-Topic Fanout, it should have little to do with the QPS for Timeline fanouts. I'd imagine the Firehose footprint is a really small part of their architecture or throughput pains. Timeline is a per-user join on multiple graphs and with paging.