TweetRex launched: Twitter friend recommender built on the Directed Edge engine

#1

Post by wheels » Tue, Mar 24, 2009, 7:53 AM UTC

TweetRex launched: Twitter friend recommender built on the Directed Edge engine
blog.directededge.com

Re: TweetRex launched: Twitter friend recommender built on the Directed Edge engine

#2

Post by uuilly » Tue, Mar 24, 2009, 8:25 AM UTC

Must be flooded. It hung for me for a while...

Re: TweetRex launched: Twitter friend recommender built on the Directed Edge engine

#3

Post by wheels » Tue, Mar 24, 2009, 8:26 AM UTC

uuilly wrote: post #2

Must be flooded. It hung for me for a while...

It takes a while to pull the data from Twitter. It actually runs around 500 worker threads spidering Twitter, and once the count gets up high some of the requests start timing out, so yeah, the whole process takes a bit.

Unfortunately, since it's a limitation in how fast we can pull data from Twitter, just throwing more hardware at the problem wouldn't help much. Performance may get better as we have a large portion of the Twitter graph cached.

We expected that it'd be a little rough today when it first goes up, but on usual days it should be able to handle a more casual load.

We debated if we should post it here since we were pretty sure it'd swamp the connection, but probably things will level out a bit by the end of the day.

Edit: Now it's hitting the VM pretty hard since once all of those threads are actually pulling, well, data, things slow down. I may set up another VPS with more RAM here in a bit and do a DNS swaperoo. We'll see how things go.

Re: TweetRex launched: Twitter friend recommender built on the Directed Edge engine

#4

Post by hajrice » Tue, Mar 24, 2009, 8:47 AM UTC

This really encourages my team and me to continue developing a major engine that'll parse content. It's so awesome that this product was built, but I'm not sure how popular it'll be. From my perspective, it has extreme pottential.

Re: TweetRex launched: Twitter friend recommender built on the Directed Edge engine

#5

Post by wheels » Tue, Mar 24, 2009, 10:24 AM UTC

Up on a second server now:

http://tweetrextoo.directededge.com/

Re: TweetRex launched: Twitter friend recommender built on the Directed Edge engine

#6

Post by apgwoz » Tue, Mar 24, 2009, 11:31 AM UTC

There's an OAuth beta now. I'll wait until they make use of it.

Re: TweetRex launched: Twitter friend recommender built on the Directed Edge engine

#7

Post by dhimes » Tue, Mar 24, 2009, 11:55 AM UTC

When I clicked someone to follow, then came back to your page (on too), I had to log in again. I was away for only moments.

I selected don't touch my stream* because I had already tweeted.

Re: TweetRex launched: Twitter friend recommender built on the Directed Edge engine

#8

Post by sandGorgon » Tue, Mar 24, 2009, 12:33 PM UTC

how does directed-edge compare to LinkedIn's OSS Project Voldemort (http://project-voldemort.com/ >) ?

Re: TweetRex launched: Twitter friend recommender built on the Directed Edge engine

#9

Post by grahamg » Tue, Mar 24, 2009, 1:29 PM UTC

Twitter is fucking stupid. Anyone who disagrees with me is a douchebag.

Re: TweetRex launched: Twitter friend recommender built on the Directed Edge engine

#10

Post by champion » Tue, Mar 24, 2009, 2:31 PM UTC

Cool, looks nice. There are several other friend recommenders out there: whoshouldifollow.com (which I helped build), Twubble, MrTweet.

I'd be curious to know more about the recommendation engine -- my recommendations skewed toward popular entities like "Whole Foods", "Zappos" and "DiningFever". Interestingly, although I follow a lot of tech/web/rails people, I didn't see any obvious folks in my recommendations.