DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second

aespinoza wrote: post #3

[deleted] wrote: post #2

[deleted]

I find it interesting, specially on what tools/languages they use, and how their architecture is designed to process such a big number of tweets.

Ugh, sorry, I finished reading right at "Information Sources", thinking that was the end of the article, and missed everything after that. I'll delete my previous comment, thanks.

Re: DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second

#5

Post by geuis » Wed, Nov 30, 2011, 12:43 AM UTC

I find the architecture to be really interesting and useful to learn from. However, they are way too expensive. 1000 tweets is not very much data. I'm building a realtime app now and easily am processing tens of thousands of relevant tweets a day. While a service like Datasift could alleviate a lot of heavy lifting on my part, the cost just doesn't make up for it. It feels like their business model is currently focused on use-cases requiring highly specific targeting, but not intended for use where services need high volumes of certain types of data. Shame, that.

Re: DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second

#6

Post by alexro » Wed, Nov 30, 2011, 1:07 AM UTC

We have recently seen a number of startups (including YC founded AFAIK) that look for ways to make twitter data useful. But I don't remember catching any notice of such successes.

While DataSift realtime capabilities look really impressive, I'm afraid there isn't that much of use-cases to pay for the data mined that way. Even DataSift's own list of possible use cases looks bleak.

In any way though DataSift should be fine with applying their expertise to other sources of data, which doesn't bear the same cost as the twitter's firehose.

Re: DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second

#7

Post by djb_hackernews » Wed, Nov 30, 2011, 1:40 AM UTC

Excellent article.

I worked on a project that I integrated with DataSift.

Lorenzo and every one else I emailed with were very quick to respond and the product performed as expected (besides some hiccups that I can easily understand due to growing pains).

Compared to Gnip, (which we also integrated with) DataSift won hands down on both quality of product and customer relations.

However, I'm still suspect of the usefulness of Twitter analytics/ data mining.

Re: DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second

#8

Post by nirvana » Wed, Nov 30, 2011, 3:02 AM UTC

geuis wrote: post #5

I find the architecture to be really interesting and useful to learn from. However, they are way too expensive. 1000 tweets is not very much data. I'm building a realtime app now and easily am processing tens of thousands of relevant tweets a day. While a service like Datasift could alleviate a lot of heavy lifting on my part, the cost just doesn't make up for it. It feels like their business model is currently focus…

Where are you getting your tweet feed? My initial interest in data sift was because they let you get a feed from the firehose. Twitter doesn't seem to let you do this.

All I want is tweets relevant to a particular subject, and in the early days I don't want to be paying hundreds of dollars for it... I've got keywords and phrases I can use to find them, if I just had access to an API that would let me. (Maybe twitter offers this, I couldn't find it in the past.)

Re: DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second

#9

Post by Titanous » Wed, Nov 30, 2011, 3:31 AM UTC

nirvana wrote: post #8

geuis wrote: post #5

I find the architecture to be really interesting and useful to learn from. However, they are way too expensive. 1000 tweets is not very much data. I'm building a realtime app now and easily am processing tens of thousands of relevant tweets a day. While a service like Datasift could alleviate a lot of heavy lifting on my part, the cost just doesn't make up for it. It feels like their business model is currently focus…

Where are you getting your tweet feed? My initial interest in data sift was because they let you get a feed from the firehose. Twitter doesn't seem to let you do this. All I want is tweets relevant to a particular subject, and in the early days I don't want to be paying hundreds of dollars for it... I've got keywords and phrases I can use to find them, if I just had access to an API that would let me. (Maybe twitter…

If you are just looking for keyword streams, they're available for free: https://dev.twitter.com/docs/streaming-api/methods

Re: DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second

#10

Post by geuis » Wed, Nov 30, 2011, 3:43 AM UTC

nirvana wrote: post #8

geuis wrote: post #5

I find the architecture to be really interesting and useful to learn from. However, they are way too expensive. 1000 tweets is not very much data. I'm building a realtime app now and easily am processing tens of thousands of relevant tweets a day. While a service like Datasift could alleviate a lot of heavy lifting on my part, the cost just doesn't make up for it. It feels like their business model is currently focus…

Where are you getting your tweet feed? My initial interest in data sift was because they let you get a feed from the firehose. Twitter doesn't seem to let you do this. All I want is tweets relevant to a particular subject, and in the early days I don't want to be paying hundreds of dollars for it... I've got keywords and phrases I can use to find them, if I just had access to an API that would let me. (Maybe twitter…

I'm using the stream, but using track filtering. So far it's working very well for my purposes. You are right in that different use cases might need the firehouse and that's where services like gnip and datasift really come in handy. It's too bad that there's not a middle ground.