Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Architecture Twitter Uses to Deal with 150M Active Users

highscalability.com

Post reply on HN
Print view View original

61–70 of 166 posts

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#61

Post by nkallen »

spullara wrote: post #34
papsosouid wrote: post #21

Earlier quoted context omitted.

The fact that you think "consumer scale web traffic" is some magical thing is exactly what I am talking about. Have you ever heard of TPC? They have done benchmarks of database driven systems for a long time. TPC-C measures performance of a particular write query, while maintaining the set ratio of other active queries. The top non-clustered result right now does 142,000 new orders per second. Yes, a single server ca…

I appreciate that you believe what you are saying, but TPC-C doesn't measure anything at all relevant. Having worked both on the enterprise side building WebLogic and on the consumer side at Yahoo and Twitter I can tell you definitively that you are wrong about the applicability of that benchmark to serving large scale web applications. The database and server you are talking about could not do 300k timeline joins pe…

sam don't waste your time this is a joke

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#62

Post by dxbydt »

ariwilson wrote: post #56
papsosouid wrote: post #50

Earlier quoted context omitted.

We're not talking about cars. We're talking about computers. 8TB of RAM is 8TB of RAM, it doesn't get better by spreading it across a thousand servers. 4096 CPU cores are 4096 CPU cores, they don't get better by spreading them across 1000 servers. Those things get worse spreading them across servers, because you massively increase the latency to access them, and for them to access shared data.

Please give an example of this "monster server" you keep talking about with 8 TB of RAM, 4096 cores, N network interface cards, and 100 TB of SSD, with a cost estimate. Otherwise we can't have a real discussion. People have a pretty good idea of how to build / what it costs to build something with 128 GB of RAM, 32 cores, a couple of NICs, and 2 TB of SSD, but what you're talking about is 50-100x beyond that.

just fyi, 2.8 million for 4096 core 16TB http://news.cnet.com/8301-30685_3-20019153-264.html?part=rss...

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#63

Post by colinhowe »

dxbydt wrote: post #62
ariwilson wrote: post #56

Earlier quoted context omitted.

Please give an example of this "monster server" you keep talking about with 8 TB of RAM, 4096 cores, N network interface cards, and 100 TB of SSD, with a cost estimate. Otherwise we can't have a real discussion. People have a pretty good idea of how to build / what it costs to build something with 128 GB of RAM, 32 cores, a couple of NICs, and 2 TB of SSD, but what you're talking about is 50-100x beyond that.

just fyi, 2.8 million for 4096 core 16TB http://news.cnet.com/8301-30685_3-20019153-264.html?part=rss...

5.6 million. You need a hot standby. Or 8.4 if you're feeling cautious :)

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#64

Post by mythz »

papsosouid wrote: post #29
jerf wrote: post #28

Earlier quoted context omitted.

I'm pretty sure the insides of your 2 $1,000,000 servers are architecturally essentially identical to a large cluster of off-the-shelf computers, with somewhat different bandwidth characteristics, but not enough to make a difference. They'll have some advanced features for making sure the $1,000,000 machine doesn't fall over. But it's not like they're magically equipped with disks that are a hundred times faster or a…

I'm not sure what you are trying to say. My entire point was that you can buy a single server that is more powerful than their entire cluster. You appear to agree, but think that is a problem?

"you can buy a single server that is more powerful than their entire cluster." is pure genius.

We now know how to solve the C10k problem in orders of magnitude - use a single server!

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#65

Post by nasalgoat »

davidw wrote: post #52

> Your home timeline sits in a Redis cluster and has a maximum of 800 entries. Wow, that's pretty cool. Congrats to antirez - it must be a nice feeling knowing that your software powers such a big system!

Redis is probably the most useful tool powering the internet after nginx. It really is an amazing piece of engineering.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#66

Post by ampersandy »

No one else seems to have noticed that these details are out of date. Twitter has publicly stated that they currently have "well over 200M active users". The stats are also misleading in that -- I'm pretty sure -- 300k reads and 6k writes per second are only referencing tweets. Flock, Twitter's graph database, handles more than 20k writes and 100k reads per second on its own (peak numbers available from the two year old Readme on Github).

https://blog.twitter.com/2013/celebrating-twitter7

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#68

Post by scott_s »

papsosouid wrote: post #29
jerf wrote: post #28

Earlier quoted context omitted.

I'm pretty sure the insides of your 2 $1,000,000 servers are architecturally essentially identical to a large cluster of off-the-shelf computers, with somewhat different bandwidth characteristics, but not enough to make a difference. They'll have some advanced features for making sure the $1,000,000 machine doesn't fall over. But it's not like they're magically equipped with disks that are a hundred times faster or a…

I'm not sure what you are trying to say. My entire point was that you can buy a single server that is more powerful than their entire cluster. You appear to agree, but think that is a problem?

Massively parallel single-machine supercomputers are still, essentially, distributed systems on the inside. You still have to use many of the same techniques to avoid communicating all-to-all. If you treat such a system as a flat memory hierarchy, your application will fall down.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#69

Post by smandou »

nasalgoat wrote: post #65
davidw wrote: post #52

> Your home timeline sits in a Redis cluster and has a maximum of 800 entries. Wow, that's pretty cool. Congrats to antirez - it must be a nice feeling knowing that your software powers such a big system!

Redis is probably the most useful tool powering the internet after nginx. It really is an amazing piece of engineering.

You forgot the JVM.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#70

Post by nasalgoat »

papsosouid wrote: post #21
spullara wrote: post #14

Earlier quoted context omitted.

I hope this comment is a troll and you don't actually believe that you could run Twitter on a single server. If you are interested in why this is true I suggest you get a job at a company that has to serve consumer scale web traffic.

The fact that you think "consumer scale web traffic" is some magical thing is exactly what I am talking about. Have you ever heard of TPC? They have done benchmarks of database driven systems for a long time. TPC-C measures performance of a particular write query, while maintaining the set ratio of other active queries. The top non-clustered result right now does 142,000 new orders per second. Yes, a single server ca…

One of the people I work with was at eBay from 2000 to 2009 and they did what you talk about - have one giant database server. Some Sun monstrosity.

Then they ran out of horsepower on that one server. Let me tell you, the stories are pretty horrible.

Post reply on HN