Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Architecture Twitter Uses to Deal with 150M Active Users

highscalability.com

Post reply on HN
Print view View original

121–130 of 166 posts

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#121

Post by jacques_chester »

Figuratively, Twitter have switched from doing a design rooted in Databases 101 (OLTP) to a design more rooted in Databases 102 (OLAP).

That is, they moved processing from query time to write time. And that's a perfectly legitimate strategy; it's the basis of data warehousing.

OLTP is about write-time speed. It's great for stuff like credit card transactions, where you only really care about the tally at the end of the banking day. The dominant mode of operation is writing.

OLAP is about read-time speed. You do a bunch of upfront processing to turn your data into something that can be queried quickly and flexibly.

One thing that's problematic about the teaching of database technology is that the read/write balance isn't explicitly taught. Your head is filled with chapter and verse of relational algebra, which is essential for good OLTP design. But the problems of querying large datasets is usually left for a different course, if it's treated at all.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#122

Post by Yhippa »

aroman wrote: post #23

> Twitter no longer wants to be a web app. Twitter wants to be a set of APIs that power mobile clients worldwide, acting as one of the largest real-time event busses on the planet. Wait, then why are they actively destroying their third-party app ecosystem...?

Probably their mobile clients. After I read the article perhaps Twitter felt that having control of the clients reduces variability in stress to their system. Like bad actors accessing the firehose suboptimally.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#123

Post by jebblue »

>> it can take up to 5 minutes for a tweet to flow from Lady Gaga’s fingers to her 31 million followers

Why not break the load up among a farm of servers? 5 minutes to deliver a single message? It's too bad multicast can't be made to work for this use case.

At least analyze to see if there's a pattern of geographic concentration of her followers and optimize for where their datacenters are.

Use peer to peer, let the clients help distribute the messages.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#124

Post by apaprocki »

jacques_chester wrote: post #118
apaprocki wrote: post #114

Earlier quoted context omitted.

Securities work a similar same way -- When you're looking at your portfolio you're only directly monitoring the individual fields for the securities in your portfolio, not the entire firehose. The OPRA feed is an equivalent to the raw Twitter firehose, only much fatter. Once you integrate it into the last-mile display to the users, you're doing all the same things. This is typically done with multicast topic-style su…

Securities feeds are, I suspect, more write-dominant and have a much lower fanout rate. They also don't have the outlier-retweeting-outlier or outlier-tweeting-@-outlier problems.

Yes, much more write-dominant. Apps which are built on top of the feeds can create issues, though. E.g., An obscure ticker pasted into a chat room with 500 people instantly starts monitoring in all their windows -- not just the static price (equiv to a RT), but the live feed. That would be as if A RT'd B and suddenly all of A's followers added B to their timeline. The degree to which that happens depends on how many features like that are integrated into the app.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#125

Post by norkakn »

davidgerard wrote: post #77
papsosouid wrote: post #10

I really question the current trend of creating big, complex, fragile architectures to "be able to scale". These numbers are a great example of why, the entire thing could run on a single server, in a very straight forward setup. When you are creating a cluster for scalability, and it has less CPU, RAM and IO than a single server, what are you gaining? They are only doing 6k writes a second for crying out loud.

I do consider it a pathology when tiny services, or tiny apps in a corporate structure, act like they have the problems of Google. You are not Google. You do not have Google's problems. You do not have scaling issues. For you, N is small and will stay small . Stop giving me this delusional resume-padding garbage to implement. For you, here, it is delusion and lies.

I remember asking a question about scaling on SO and getting a response about how my data was small, because it could fit on an SSD (for that one small component)

I miss not having real scaling issues sometimes. They make my head hurt.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#126

Post by joshuaellinger »

jacques_chester wrote: post #121

Figuratively, Twitter have switched from doing a design rooted in Databases 101 (OLTP) to a design more rooted in Databases 102 (OLAP). That is, they moved processing from query time to write time. And that's a perfectly legitimate strategy; it's the basis of data warehousing. OLTP is about write-time speed. It's great for stuff like credit card transactions, where you only really care about the tally at the end of t…

Maybe at a high level, but OLAP involves precomputing aggregations and bit indexes. It's a pretty different beast.

OLAP is very rarely under real-time constraints and, when it is, it tends to push the heavy lifting out to OLTP.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#127

Post by joshuaellinger »

The surprise for me is that the core component is Redis.

My first guess would have been custom C code. Yeah, you have to do everything yourself. Yeah, it would be hard to write. But you'd control every little bit of it.

Obviously, I must not fully understand the problem and what Redis buys.

Sam Puralla (if you are reading) -- do you know why didn't Twitter go with a full custom system at its heart?

Josh

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#128

Post by jacques_chester »

joshuaellinger wrote: post #126
jacques_chester wrote: post #121

Figuratively, Twitter have switched from doing a design rooted in Databases 101 (OLTP) to a design more rooted in Databases 102 (OLAP). That is, they moved processing from query time to write time. And that's a perfectly legitimate strategy; it's the basis of data warehousing. OLTP is about write-time speed. It's great for stuff like credit card transactions, where you only really care about the tally at the end of t…

Maybe at a high level, but OLAP involves precomputing aggregations and bit indexes. It's a pretty different beast. OLAP is very rarely under real-time constraints and, when it is, it tends to push the heavy lifting out to OLTP.

Oh, I'm handwaving, not talking about the underlying details of star schemata, clever column representations and whatnot.

But I think the analogy is still correct.

Every such system has two functional requirements:

1. Store data.

2. Query data.

And every system has the same non-functional requirement:

1. Storage (write) should be as fast as possible.

2. Queries (read) should be as fast as possible.

However, per an observation I made a while back, complexity in the problem domain is a conserved value.

Insofar as your data requires processing to be useful, that complexity cannot be made to go away. You can only decide where the complexity cost is paid.

You can pay it at write time and amortise that across reads. You can pay it at read time and excuse writes. Or you can pay it in the middle with some sort of ETL pipeline or processing queue.

But you must always pay it. The experiences of data warehousing made that bitterly clear.

So really, the job of a software architect is to take the business requirements as a non-functional requirement (an -ility) and then pick the architecture that fits that NFR. That includes dropping other nice-to-have non-functionals.

Twitter's non-functional requirement is that they want end-to-end latency to be 5 seconds or less, under conditions of very low write:read ratio. This suggests paying the complexity cost up front and amortising it over reads. And that's what they've done.

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#129

Post by nikcub »

joshuaellinger wrote: post #127

The surprise for me is that the core component is Redis. My first guess would have been custom C code. Yeah, you have to do everything yourself. Yeah, it would be hard to write. But you'd control every little bit of it. Obviously, I must not fully understand the problem and what Redis buys. Sam Puralla (if you are reading) -- do you know why didn't Twitter go with a full custom system at its heart? Josh

Think of Redis as a nicer interface to data structures that otherwise would have been written in custom C code.

If you read the redis manifesto, it describes itself in point 1 as a DSL for abstract data types:

http://oldblog.antirez.com/post/redis-manifesto.html

Re: The Architecture Twitter Uses to Deal with 150M Active Users

#130

Post by squarecog »

jacques_chester wrote: post #121

Figuratively, Twitter have switched from doing a design rooted in Databases 101 (OLTP) to a design more rooted in Databases 102 (OLAP). That is, they moved processing from query time to write time. And that's a perfectly legitimate strategy; it's the basis of data warehousing. OLTP is about write-time speed. It's great for stuff like credit card transactions, where you only really care about the tally at the end of t…

That's an interesting thought, but rather than focusing on write vs read balance, I would claim that OLAP and OLTP are most distinguished by the nature of queries they need to support.

OLAP is characterized by fairly low-volume aggregation queries that touch very large volumes of data ("what fraction of tweets in English come from non-english speaking countries?"). Sure, the ingest is large, but it tends to be batch and has fairly loose real-time (in the EE sense) requirements. What makes OLAP hard is the sheer volume of data from which an aggregation must be calculated.

OLTP is characterized by very selective queries with much tighter real-time bounds ("what are the last 20 things that the 100 people I follow said?"). The overall size of the dataset might even be the same, but each individual query needs fast access to a tiny fraction of the dataset. In many applications, this is accompanied by very high QPS and in Twitter's case, extremely high write volume.

Post reply on HN