The Architecture Twitter Uses to Deal with 150M Active Users • Hacker News Board

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#41

Post by rythie » Mon, Jul 08, 2013, 7:56 PM UTC

papsosouid wrote: post #40

rythie wrote: post #37

Earlier quoted context omitted.

$1,000,000 machines are not that much faster, you won't get 1000x the performance or anywhere near. The memory and IO performance is going to be within a factor 2 or 4 of that high end machine. It might have 50x the cores, but most likely that's not the limiting factor anyway.

The whole point is you can get equal performance from a single server instead of a ton of little ones. The ton of little ones forced them to totally re-architect to work around the massive latency between servers. A single server would have allowed them to stick with a sane architecture, and saved them millions in development time and maintenance nightmares.

It would be nice if were true, but you simply can't, there's no magic that makes an expensive server that much faster - it's just a bit faster for a lot more money. It can make sense if you only want 10x the performance and the server is cheaper than the rewrite.

For example, if a $30k car can go 150mph, it doesn't mean a $300k car can go 1,500mph it just doesn't happen. A Bugatti Veyron goes, what? 254mph that's not even double (and it costs a lot more than $300k)

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#42

Post by adient » Mon, Jul 08, 2013, 7:57 PM UTC

papsosouid wrote: post #10

I really question the current trend of creating big, complex, fragile architectures to "be able to scale". These numbers are a great example of why, the entire thing could run on a single server, in a very straight forward setup. When you are creating a cluster for scalability, and it has less CPU, RAM and IO than a single server, what are you gaining? They are only doing 6k writes a second for crying out loud.

You're arguing for scaling up (vertical) instead of scaling out (horizontal). Both are valid approaches. Scaling out is preferred because your architecture is mode modular and you do not have to constantly buy bigger machines as your usage grows; you just add additional machines with similar capacity. The main problem with scaling up is that price and performance are not linearly related and eventually you will be limited by the performance available to one system. But it's a perfectly valid approach for certain scenarios.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#43

Post by numbsafari » Mon, Jul 08, 2013, 8:01 PM UTC

papsosouid wrote: post #10

I really question the current trend of creating big, complex, fragile architectures to "be able to scale". These numbers are a great example of why, the entire thing could run on a single server, in a very straight forward setup. When you are creating a cluster for scalability, and it has less CPU, RAM and IO than a single server, what are you gaining? They are only doing 6k writes a second for crying out loud.

Because if you want to have reasonable availability and low latency you can't just buy two "big" machines.

Going with the incrementally sized approach gives you more flexibility to economically distribute the risk, in addition to the load.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#44

Post by blazingfrog2 » Mon, Jul 08, 2013, 8:07 PM UTC

aroman wrote: post #23

> Twitter no longer wants to be a web app. Twitter wants to be a set of APIs that power mobile clients worldwide, acting as one of the largest real-time event busses on the planet. Wait, then why are they actively destroying their third-party app ecosystem...?

The piece did not say "third party". I'm assuming these clients are the official Twitter mobile apps.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#45

Post by OGC » Mon, Jul 08, 2013, 8:09 PM UTC

papsosouid wrote: post #24

nostrademons wrote: post #20

Earlier quoted context omitted.

Do you hit the disk when somebody, say, checks my Twitter profile that I haven't updated since 2008? What will that do to your performance?

I'm not sure why you are asking me about this.

Because he/she hasn't read the article.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#46

Post by thezilch » Mon, Jul 08, 2013, 8:10 PM UTC

papsosouid wrote: post #10

I really question the current trend of creating big, complex, fragile architectures to "be able to scale". These numbers are a great example of why, the entire thing could run on a single server, in a very straight forward setup. When you are creating a cluster for scalability, and it has less CPU, RAM and IO than a single server, what are you gaining? They are only doing 6k writes a second for crying out loud.

They may average 6K w/s, but history has shown higher peaks, and those peaks often include tweets from those with large 30e6+ followers -- large events naturally have celebrities chiming in. Having more CPU [cores], RAM, and IO is not going to solve for the need to shard your Redis, which is bound by a single core. For the fanout, that is anywhere from 90 to 90e3 cores at 2e6 Redis IOPs -- 1-6e3 Lady Gagas per second. Before pipelining, which you'd probably consider complex or fragile. All you've done is moved some networking parts around. Your code and topology hasn't really changed.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#47

Post by brown9-2 » Mon, Jul 08, 2013, 8:13 PM UTC

mikemoka wrote: post #26

I am playing the armchair architect and my question will be probably wrong in infinite ways,but I might learn something, what is the reason why the service has to write a tweet on two million timelines, wouldn't it be cheaper if they let the client build the page on its own via restful apis?

They are making a tradeoff between cost of reading your timeline and cost of a write at tweet-time to optimize the former.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#48

Post by brown9-2 » Mon, Jul 08, 2013, 8:15 PM UTC

papsosouid wrote: post #10

I really question the current trend of creating big, complex, fragile architectures to "be able to scale". These numbers are a great example of why, the entire thing could run on a single server, in a very straight forward setup. When you are creating a cluster for scalability, and it has less CPU, RAM and IO than a single server, what are you gaining? They are only doing 6k writes a second for crying out loud.

They are only doing 6k writes a second for crying out loud.

And 600k reads per second (as of when this article was written), which seems like an important thing to leave out.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#49

Post by binarycrusader » Mon, Jul 08, 2013, 8:30 PM UTC

bpicolo wrote: post #2

They have a lot of RAM. Dang.

That's really not a lot, especially when you consider that it's distributed among a cluster.

A single SPARC M5-32 can have up to 32TB of memory:

http://www.oracle.com/us/products/servers-storage/servers/sp...

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#50

Post by papsosouid » Mon, Jul 08, 2013, 8:35 PM UTC

rythie wrote: post #41

papsosouid wrote: post #40

Earlier quoted context omitted.

The whole point is you can get equal performance from a single server instead of a ton of little ones. The ton of little ones forced them to totally re-architect to work around the massive latency between servers. A single server would have allowed them to stick with a sane architecture, and saved them millions in development time and maintenance nightmares.

It would be nice if were true, but you simply can't, there's no magic that makes an expensive server that much faster - it's just a bit faster for a lot more money. It can make sense if you only want 10x the performance and the server is cheaper than the rewrite. For example, if a $30k car can go 150mph, it doesn't mean a $300k car can go 1,500mph it just doesn't happen. A Bugatti Veyron goes, what? 254mph that's not…

We're not talking about cars. We're talking about computers. 8TB of RAM is 8TB of RAM, it doesn't get better by spreading it across a thousand servers. 4096 CPU cores are 4096 CPU cores, they don't get better by spreading them across 1000 servers. Those things get worse spreading them across servers, because you massively increase the latency to access them, and for them to access shared data.