Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)

techcrunch.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)

#21

Post by ajays »

I'm missing the point, so I hope the HN community can enlighten me, but: why do you need Cassandra and all that jazz, if you're just incrementing counters?

A decent memcached instance on modern hardware can easily push several 100K updates/sec . Couldn't you do the same with a pool of sharded memcached servers? Compute the MD5 hash of the string you want to count (which can be 16 strings, from the top-rated comment above), and just use that as the key.

We've seen people push 750K QPS on an InnoDB via HandlerSockets http://news.ycombinator.com/item?id=1886137 , so imagine what you could do with a sharded pool of 20 InnoDB servers.

Again: if I'm missing something, I'd love to learn.

Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)

#23

Post by afex »

ajays wrote: post #21

I'm missing the point, so I hope the HN community can enlighten me, but: why do you need Cassandra and all that jazz, if you're just incrementing counters? A decent memcached instance on modern hardware can easily push several 100K updates/sec . Couldn't you do the same with a pool of sharded memcached servers? Compute the MD5 hash of the string you want to count (which can be 16 strings, from the top-rated comment a…

As far as datastores go, cassandra is write-optimized. Writes are faster than reads. So for this use case (heavy denormalization) it is a good fit.

Also, Cassandra reduces the operational complexity of having a logical store which spans multiple hosts. Your memcache example does not get persistence for free, and sharding mysql is something you have to do manually. The interface to a Cassandra cluster is the same regardless of how many nodes you are running.

Scaling writes is "hard". Incrementing counters is obviously write heavy, and cassandra aims to make it easier.

Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)

#24

Post by saurik »

ajays wrote: post #21

I'm missing the point, so I hope the HN community can enlighten me, but: why do you need Cassandra and all that jazz, if you're just incrementing counters? A decent memcached instance on modern hardware can easily push several 100K updates/sec . Couldn't you do the same with a pool of sharded memcached servers? Compute the MD5 hash of the string you want to count (which can be 16 strings, from the top-rated comment a…

The day half your memcached data center loses power will be a sad sad day indeed, and the story of 750k QPS on InnoDB was about read traffic, not writes.

Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)

#25

Post by JoachimSchipper »

jhugg wrote: post #5

Shameless self-promotion, but important point nonetheless: VoltDB is truly fantastic at this kind of workload. It's one of the first use cases we had traction with. Also, VoltDB is open source today.

I didn't know VoltDB was open source, actually. Cool!

Is the database really entirely in-memory, though?

Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)

#26

Post by lsb »

jhugg wrote: post #5

Shameless self-promotion, but important point nonetheless: VoltDB is truly fantastic at this kind of workload. It's one of the first use cases we had traction with. Also, VoltDB is open source today.

VoltDB looks pretty awesome, but I'm pretty concerned by its lack of ability to join a table to itself, to have over six tables in a join, to aggregate or select distinct arbitrary values. http://community.voltdb.com/docs/ReleaseNotes/index

Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)

#27

Post by moe »

trafficlight wrote: post #13
mmaunder wrote: post #10

Twitter is clearly brilliant and creating a viral and useful product. I admire everything they've achieved in terms of user adoption and usefulness. They're proven without a doubt that technology is only one ingredient and it doesn't have to work well for a web or mobile business to grow. They're the last company I'll look to for technology to use in my business or as an example on how to run operations. I have no in…

On what basis? Because they've been through some rough patches where they couldn't keep up with the traffic, you'll dismiss everything they've built?

I agree with his general sentiment, I'd only phrase it differently.

The point is not that they've been "through some rough patches". The point is that they failed for years to come up with a reliable implementation of a solved problem; pub/sub messaging.

Twitter is not "large" by any means. Your telco, stock exchange and many other companies have dealt with the the same problem-space for decades. Those reliably dispatch orders of magnitude higher throughput under much more complex routing conditions. Many of them operate under SLAs that mandate five or even six nines of availability.

Sure, twitter is (gladly) not dispatching emergency-calls, as such their requirements are lower. However, given their track-record they're in no position to give technology advice either.

50 million tweets/day[1] is not a serious workload for a messaging system.

http://mashable.com/2010/02/22/twitter-50-million-tweets/

Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)

#28

Post by jhugg »

lsb wrote: post #26
jhugg wrote: post #5

Shameless self-promotion, but important point nonetheless: VoltDB is truly fantastic at this kind of workload. It's one of the first use cases we had traction with. Also, VoltDB is open source today.

VoltDB looks pretty awesome, but I'm pretty concerned by its lack of ability to join a table to itself, to have over six tables in a join, to aggregate or select distinct arbitrary values. http://community.voltdb.com/docs/ReleaseNotes/index

At VoltDB, we're working on improving our SQL support. Initially, we focused on core SQL useful for OLTP. We've added functionality with every release and we plan to continue in 2011.

For example, the March release of VoltDB will support more than 6 table joins and has some improvements to aggregation and distinct code. It also has a much more usable explain plan feature.

We feel that VoltDB offers one of the richer query interfaces of systems that scale to its level, but we don't plan to sit still.

Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)

#29

Post by jhugg »

JoachimSchipper wrote: post #25
jhugg wrote: post #5

Shameless self-promotion, but important point nonetheless: VoltDB is truly fantastic at this kind of workload. It's one of the first use cases we had traction with. Also, VoltDB is open source today.

I didn't know VoltDB was open source, actually. Cool! Is the database really entirely in-memory, though?

VoltDB relies on syncronous replication to protect data from hardware and network failures.

It also supports continuous and transactionally-isolated snapshotting to disk.

Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)

#30

Post by [deleted] »

saurik wrote: post #24
ajays wrote: post #21

I'm missing the point, so I hope the HN community can enlighten me, but: why do you need Cassandra and all that jazz, if you're just incrementing counters? A decent memcached instance on modern hardware can easily push several 100K updates/sec . Couldn't you do the same with a pool of sharded memcached servers? Compute the MD5 hash of the string you want to count (which can be 16 strings, from the top-rated comment a…

The day half your memcached data center loses power will be a sad sad day indeed, and the story of 750k QPS on InnoDB was about read traffic, not writes.

[deleted]
Post reply on HN