I particularly liked the first line on slide 48
"We require operational reporting on all services"
feels like everyone learns that the hard way, twitter just did it much more visibly
Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
11–20 of 31 posts
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#12Twitter is clearly brilliant and creating a viral and useful product. I admire everything they've achieved in terms of user adoption and usefulness. They're proven without a doubt that technology is only one ingredient and it doesn't have to work well for a web or mobile business to grow. They're the last company I'll look to for technology to use in my business or as an example on how to run operations. I have no in…
[deleted]
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#13Twitter is clearly brilliant and creating a viral and useful product. I admire everything they've achieved in terms of user adoption and usefulness. They're proven without a doubt that technology is only one ingredient and it doesn't have to work well for a web or mobile business to grow. They're the last company I'll look to for technology to use in my business or as an example on how to run operations. I have no in…
On what basis? Because they've been through some rough patches where they couldn't keep up with the traffic, you'll dismiss everything they've built?
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#14Twitter is clearly brilliant and creating a viral and useful product. I admire everything they've achieved in terms of user adoption and usefulness. They're proven without a doubt that technology is only one ingredient and it doesn't have to work well for a web or mobile business to grow. They're the last company I'll look to for technology to use in my business or as an example on how to run operations. I have no in…
I honestly did not understand if you were being sarcastic in your third paragraph.
I'm still not sure, but it seems you're peeved that Twitter hasn't been perfect and not impressed with Rainbird. (Right?)
This is a bit unfair I think? No one's really waiting for your advice on how to run your data center, whereas Twitter is dealing with the sort of volume that your data center probably couldn't begin to handle without finding some of the same solutions Twitter is now presenting.
Now we could possibly (depending on what you actually meant) get into a debate about Twitter's failures, but that's really not interesting, because your second paragraph is true: technological perfection is not enough by itself.
That doesn't mean Twitter doesn't have anything interesting to say about technological perfection.
But mostly I posted this in order to express the sheer confusion I felt reading your post. If you don't like Twitter, just say so.
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#15"twitter try-hard or what...
The very existence of that "insult" amuses me. I miss the days when cooler than cool meant ice cold.
I posted this partially to experimentally examine the downvote clumping effect I've noticed on hn, where some reasonable responses to highly downvoted posts get downvoted by association.
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#16On another note, last year Twitter open sourced Gizzard.
http://engineering.twitter.com/2010/04/introducing-gizzard-f... (ironically, nobody seems to have tweeted this article!!)
Natural curiosity-I downloaded the repo,and tried to understand what it was.
Later,it also seemed the buzz surrounding ( on HN atleast) didnt last much, and I forgot about Gizzard completely.
Are there startups/bigCos using Gizzard,other Twitter's open source stuff?
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#17meh? ultimately the back end for it is cassandra, correct? so they're what, going to open source their implementation of doing stats in cassandra? It's great for people to open source code, but I think it's more of a championing for cassandra, than it is a 'code dump'
also in that vein, the formspring comment / gist link is very commendable and awesome!
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#18Shameless self-promotion, but important point nonetheless: VoltDB is truly fantastic at this kind of workload. It's one of the first use cases we had traction with. Also, VoltDB is open source today.
I think that at the rates Twitter is writing counter data (Many TBs per day denormalized, ~0.5TB normalized), a RAM-based solution like VoltDB would be prohibitively expensive. Rainbird allows Twitter to use cheap disk-based storage but still get acceptable (sub-second) latency.
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#19Shameless self-promotion, but important point nonetheless: VoltDB is truly fantastic at this kind of workload. It's one of the first use cases we had traction with. Also, VoltDB is open source today.
I think that at the rates Twitter is writing counter data (Many TBs per day denormalized, ~0.5TB normalized), a RAM-based solution like VoltDB would be prohibitively expensive. Rainbird allows Twitter to use cheap disk-based storage but still get acceptable (sub-second) latency.
Perhaps, but it really depends on what you want to do exactly. You can probably store a lot less and get the same throughput.
- VoltDB isn't log-structured, so you really only have to store the state. How fast you can mutate it isn't limited by RAM amounts. We see use cases with utter firehoses of data that update just tens or hundreds of gigabytes of state.
- Beyond normalization, you can probably reduce the number of redundant counters, e.g. use SQL to count which URLs start with "amazon". This would be painful in many systems, but depending on the query, can often be done at scale in VoltDB.
- The byte overhead per counter is also likely much lower in an ACID/Relational store.
Finally, VoltDB is designed to migrate data to disk based stores (such as Hadoop or an OLAP store) as memory fills up. This is a feature we're working very hard on see as a big differentiator. It adds complexity if you need to query across stores, but you get a best-of-both-worlds feature set.
Re: Rainbird: The Way Twitter Counts Tweets In Realtime (Soon To Be Open Sourced)
#20On another note, last year Twitter open sourced Gizzard. http://engineering.twitter.com/2010/04/introducing-gizzard-f... (ironically, nobody seems to have tweeted this article!!) Natural curiosity-I downloaded the repo,and tried to understand what it was. Later,it also seemed the buzz surrounding ( on HN atleast) didnt last much, and I forgot about Gizzard completely. Are there startups/bigCos using Gizzard,other Twi…
i played with gizzard for one of our services with redis as backend. twitter engineers on #twinfra IRC channel helped me a lot on understanding gizzard and flockdb src code. Not yet using in production though.
You have any questions ?