The Architecture Twitter Uses to Deal with 150M Active Users • Hacker News Board

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#81

Post by rozap » Mon, Jul 08, 2013, 10:56 PM UTC

papsosouid wrote: post #54

thezilch wrote: post #46

Earlier quoted context omitted.

They may average 6K w/s, but history has shown higher peaks, and those peaks often include tweets from those with large 30e6+ followers -- large events naturally have celebrities chiming in. Having more CPU [cores], RAM, and IO is not going to solve for the need to shard your Redis, which is bound by a single core. For the fanout, that is anywhere from 90 to 90e3 cores at 2e6 Redis IOPs -- 1-6e3 Lady Gagas per second…

>Having more CPU [cores], RAM, and IO is not going to solve for the need to shard your Redis, which is bound by a single core But they chose redis in the first place as part of their messy, crummy pile of barely working junkware solution. You wouldn't use redis in the first place if you were building a solution that scales up.

You're funny. I like you.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#82

Post by zanny » Mon, Jul 08, 2013, 10:58 PM UTC

mey wrote: post #51

aroman wrote: post #23

> Twitter no longer wants to be a web app. Twitter wants to be a set of APIs that power mobile clients worldwide, acting as one of the largest real-time event busses on the planet. Wait, then why are they actively destroying their third-party app ecosystem...?

The two are not the same. It sounds like they are becoming API driven to support multiple interfaces and to put clear boundaries between systems.

Sounds like html. Or qt. Not 140 characters.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#83

Post by thezilch » Mon, Jul 08, 2013, 11:31 PM UTC

papsosouid wrote: post #54

thezilch wrote: post #46

Earlier quoted context omitted.

They may average 6K w/s, but history has shown higher peaks, and those peaks often include tweets from those with large 30e6+ followers -- large events naturally have celebrities chiming in. Having more CPU [cores], RAM, and IO is not going to solve for the need to shard your Redis, which is bound by a single core. For the fanout, that is anywhere from 90 to 90e3 cores at 2e6 Redis IOPs -- 1-6e3 Lady Gagas per second…

>Having more CPU [cores], RAM, and IO is not going to solve for the need to shard your Redis, which is bound by a single core But they chose redis in the first place as part of their messy, crummy pile of barely working junkware solution. You wouldn't use redis in the first place if you were building a solution that scales up.

So, now we also have to assume per-core software and support licenses to accompany our purchase of several $10M racks? What platform are we talking about now? Presumably I have to also purchase the IDE and toolchain licenses, per developer, too?

Twitter may have and will still make missteps, but you're being egregiously vague and unhelpful. Your comments are as vaporous as you make Twitter's architecture to be.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#84

Post by harryh » Mon, Jul 08, 2013, 11:37 PM UTC

mikemoka wrote: post #26

I am playing the armchair architect and my question will be probably wrong in infinite ways,but I might learn something, what is the reason why the service has to write a tweet on two million timelines, wouldn't it be cheaper if they let the client build the page on its own via restful apis?

That is actually a very interesting question and it turns out that whether it's better to fanout on write or on read depends on a few different things. There's a very widely read paper on the subject you might enjoy:

http://research.yahoo.net/node/3203

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#85

Post by eaceaser » Mon, Jul 08, 2013, 11:45 PM UTC

nkallen wrote: post #61

spullara wrote: post #34

Earlier quoted context omitted.

I appreciate that you believe what you are saying, but TPC-C doesn't measure anything at all relevant. Having worked both on the enterprise side building WebLogic and on the consumer side at Yahoo and Twitter I can tell you definitively that you are wrong about the applicability of that benchmark to serving large scale web applications. The database and server you are talking about could not do 300k timeline joins pe…

sam don't waste your time this is a joke

sup nick

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#86

Post by wmf » Mon, Jul 08, 2013, 11:49 PM UTC

mwcampbell wrote: post #79

papsosouid wrote: post #10

I really question the current trend of creating big, complex, fragile architectures to "be able to scale". These numbers are a great example of why, the entire thing could run on a single server, in a very straight forward setup. When you are creating a cluster for scalability, and it has less CPU, RAM and IO than a single server, what are you gaining? They are only doing 6k writes a second for crying out loud.

Maybe Twitter would indeed be better off using a smaller number of more beefy servers. But at the software level, I wonder if the architecture would be very different. Suppose you implemented something like Twitter as a single large application, running in a single process. Maybe run the RDBMS in its own process. The app server stores a lot of stuff in its own in-process memory, instead of using Redis or the like. No…

You'd use failover to do a rolling upgrade.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#87

Post by sulam » Mon, Jul 08, 2013, 11:55 PM UTC

papsosouid wrote: post #16

colinhowe wrote: post #13

Earlier quoted context omitted.

That's probably 6k writes to a "normal" DB. The fanout is handled by Redis which I doubt is included in the 6k writes. Not if you want to fan out to 30 million followers in under 5 seconds.

The "fanout" or "wasteful duplication of a single message 30 million times" is only required because they are using tiny underpowered hardware to begin with. The approach that they claim can't possibly work actually does work. You just can't do it on a $2000 "server".

Let me guess, you work for Oracle?

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#88

Post by kingryan » Mon, Jul 08, 2013, 11:57 PM UTC

papsosouid wrote: post #16

colinhowe wrote: post #13

Earlier quoted context omitted.

That's probably 6k writes to a "normal" DB. The fanout is handled by Redis which I doubt is included in the 6k writes. Not if you want to fan out to 30 million followers in under 5 seconds.

The "fanout" or "wasteful duplication of a single message 30 million times" is only required because they are using tiny underpowered hardware to begin with. The approach that they claim can't possibly work actually does work. You just can't do it on a $2000 "server".

The messages are not repeated in fanout, just the ids. I guess you would know that if you actually the links.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#89

Post by kingryan » Mon, Jul 08, 2013, 11:59 PM UTC

papsosouid wrote: post #53

brown9-2 wrote: post #48

Earlier quoted context omitted.

They are only doing 6k writes a second for crying out loud. And 600k reads per second (as of when this article was written), which seems like an important thing to leave out.

300k reads, which is approaching trivial. You can do that with an array of desktop SSDs. But in reality, it would be almost all in RAM anyways making those reads incredibly cheap.

Right, because no matter how much RAM you put in a machine, access is always the same speed.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#90

Post by vidarh » Tue, Jul 09, 2013, 12:04 AM UTC

ariwilson wrote: post #56

papsosouid wrote: post #50

Earlier quoted context omitted.

We're not talking about cars. We're talking about computers. 8TB of RAM is 8TB of RAM, it doesn't get better by spreading it across a thousand servers. 4096 CPU cores are 4096 CPU cores, they don't get better by spreading them across 1000 servers. Those things get worse spreading them across servers, because you massively increase the latency to access them, and for them to access shared data.

Please give an example of this "monster server" you keep talking about with 8 TB of RAM, 4096 cores, N network interface cards, and 100 TB of SSD, with a cost estimate. Otherwise we can't have a real discussion. People have a pretty good idea of how to build / what it costs to build something with 128 GB of RAM, 32 cores, a couple of NICs, and 2 TB of SSD, but what you're talking about is 50-100x beyond that.

Why should he give an example of a "monster server" with specs like that?

He gave the argument that spreading the CPU's and memory out does not make them better.

So part of the point is that if the starting point is 8TB of RAM and 4096 cores distributed over a bunch of machines, then a "large machine" approach will require substantially less.

I've not done the maths for whether or not a "Twitter scale" app would fit on a single current generation mainframe or similar large scale "machine" (what constitutes a "machine" becomes nebulous since many of the high end setups are cabinets of drawers of cards and on persons "machine" is another persons highly integrated cluster), but it would need to include a discussion of how much a tighter integrated hardware system would reduce their actual resource requirements.