Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
1–10 of 31 posts
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#2First announced in Strata conference. Here is the link to the presentation- Realtime Analytics at Twitter:
http://assets.en.oreilly.com/1/event/55/Realtime%20Analytics...
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#3I was at the presentation - it's a very smart system. They basically maintain a whole bunch of counters for any particular thing that's being tracked. For example, say someone clicks on a t.co link to blog.example.com/foo at 11:41am on 1st Feb. Rainbird would increment counters for:
t.co click: com (all time)
t.co click: com.example (all time)
t.co click: com.example.blog (all time)
t.co click: com.example.blog /foo (all time)
t.co click: com (1st Feb 2011)
t.co click: com.example (1st Feb 2011)
t.co click: com.example.blog (1st Feb 2011)
t.co click: com.example.blog /foo (1st Feb 2011)
t.co click: com (11am-12 on 1st Feb)
t.co click: com.example (11am-12 on 1st Feb)
t.co click: com.example.blog (11am-12 on 1st Feb)
t.co click: com.example.blog /foo (11am-12 on 1st Feb)
t.co click: com (11:41-42 on 1st Feb)
t.co click: com.example (11:41-42 on 1st Feb)
t.co click: com.example.blog (11:41-42 on 1st Feb)
t.co click: com.example.blog /foo (11:41-42 on 1st Feb)
So that's 16 counters to track one link, but it means they can do fast, denormalised queries in realtime to track how that link is performing.
It's not just for t.co links - they can use it for internal server monitoring tools, tweet counts, advertising metrics... pretty much anything that involves counting at scale.
It's possible to build a similar system for much smaller scale applications using atomic counters in Redis - I've been experimenting with something like that for some of my own projects.
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#4Watch this talk[0] from Qcon SF November 2010. Ryan King goes into a lot of detail about many nosql implementations at twitter. This is the first time I had heard about distributed counting by way of Cassandra to implement tweet counts specifically and obviously count other things in general. In particular Cassandra is pointed out as having tremendous write availability to enable this sort of thing. He also mentions various specifics in how the feature is coded/designed and different approaches they had to take until they got a final version.
[0]http://www.infoq.com/presentations/NoSQL-at-Twitter-by-Ryan-...
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#5Shameless self-promotion, but important point nonetheless:
VoltDB is truly fantastic at this kind of workload. It's one of the first use cases we had traction with. Also, VoltDB is open source today.
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#6I was at the presentation - it's a very smart system. They basically maintain a whole bunch of counters for any particular thing that's being tracked. For example, say someone clicks on a t.co link to blog.example.com/foo at 11:41am on 1st Feb. Rainbird would increment counters for: t.co click: com (all time) t.co click: com.example (all time) t.co click: com.example.blog (all time) t.co click: com.example.blog /foo…
if you follow various twitter tech talks you'll notice that they spend a lot of time talking about 'fan out'. they basically prefer writing many things asynchronously vs. having to aggregate later. makes sense... disk is cheap and only getting cheaper. but processing time to return realtime data and information is finite.
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#7"twitter try-hard or what...
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#8I was at the presentation - it's a very smart system. They basically maintain a whole bunch of counters for any particular thing that's being tracked. For example, say someone clicks on a t.co link to blog.example.com/foo at 11:41am on 1st Feb. Rainbird would increment counters for: t.co click: com (all time) t.co click: com.example (all time) t.co click: com.example.blog (all time) t.co click: com.example.blog /foo…
We use Redis for this kind of stat tracking at http://www.formspring.me/
At 10,000,000 responses per day, we're not that small.
Here's the source from our collector: https://gist.github.com/794474
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#9That's nice but when are we going to get the deep search that will go back more than a week, which they promised over a year ago.
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#10Twitter is clearly brilliant and creating a viral and useful product. I admire everything they've achieved in terms of user adoption and usefulness.
They're proven without a doubt that technology is only one ingredient and it doesn't have to work well for a web or mobile business to grow.
They're the last company I'll look to for technology to use in my business or as an example on how to run operations. I have no interest in any of their open sourced products or in any advice from them on how to run my data center.