A new experimental "similarity search" algorithm. Thoughts?
1–10 of 14 posts
Re: A new experimental "similarity search" algorithm. Thoughts?
#2Currently, we're text-mining the english version of Wikipedia.
There's a lot of room for improvement: optimizing for speed and pruning down the results are at the top of our "TODO" list.
Also, the UI is simplistic -- that's because we've been spending 99% of our time working on the algorithm in matlab.
But, we wanted to get something out -- warts and all -- to get some feedback on the general idea.
We'd value any feedback -- positive or negative.
Re: A new experimental "similarity search" algorithm. Thoughts?
#3Currently, we're text-mining the english version of Wikipedia. There's a lot of room for improvement: optimizing for speed and pruning down the results are at the top of our "TODO" list. Also, the UI is simplistic -- that's because we've been spending 99% of our time working on the algorithm in matlab. But, we wanted to get something out -- warts and all -- to get some feedback on the general idea. We'd value any fee…
Pretty cool, what is the kind of math behind it? PLSA?
Re: A new experimental "similarity search" algorithm. Thoughts?
#4Besides the fact that it thinks too long, what is new about it, what are you trying to achieve?
I search for "Test" and results were not really relevant.
Re: A new experimental "similarity search" algorithm. Thoughts?
#5Besides the fact that it thinks too long, what is new about it, what are you trying to achieve? I search for "Test" and results were not really relevant.
Thank you for trying it out.
Re: performance - It's mining through ~40gb of data on server with 8gb of ram. - Also, we're not using caching of search results -- it computes on-the-fly for each query. - If we can get a hold of more servers, we should be able to bring down the query time below 1 second.
Re: query "Test" - You have to search for something you're interested in.
Re: A new experimental "similarity search" algorithm. Thoughts?
#6Currently, we're text-mining the english version of Wikipedia. There's a lot of room for improvement: optimizing for speed and pruning down the results are at the top of our "TODO" list. Also, the UI is simplistic -- that's because we've been spending 99% of our time working on the algorithm in matlab. But, we wanted to get something out -- warts and all -- to get some feedback on the general idea. We'd value any fee…
Pretty cool, what is the kind of math behind it? PLSA?
Yes.
At this point we're really constrained by the number of cores we're running on.
Once we can get a hold of some more servers, we should be able to drastically improve the performance and prune many of the results.
We'd also like to run the algorithm on additional corpora. Specifically: (1) the US patent database (back to 1975); and (2), a collection of United States federal and state case law (the JURIS database).
Re: A new experimental "similarity search" algorithm. Thoughts?
#7I tried Bach and it appears that it just returned Wikipedia articles that mentioned Bach. My initial expectations were more along the lines of Google Sets.
What are the shapes on the homepage for? It was kind of intriguing that they seemed to do something, but I didn't know what.
Re: A new experimental "similarity search" algorithm. Thoughts?
#8For what it's worth I searched for "google.com" and got "The International Society for Cryptozoology" as a top hit. Not sure how useful it was but I did learn that there was such a thing as Crytozoology and my day instantly got 10% better.
I actually use this one all the time: http://www.similicio.us/
I think it uses delicious tags.
Re: A new experimental "similarity search" algorithm. Thoughts?
#9I tried Bach and it appears that it just returned Wikipedia articles that mentioned Bach. My initial expectations were more along the lines of Google Sets. What are the shapes on the homepage for? It was kind of intriguing that they seemed to do something, but I didn't know what.
You have to click on the bold links to "drill-down". So, if you click on "johann sebastian bach", it will give start to give you results such as: "Vivaldi's Cello" and "History of Germany"
Currently, we're limited to running on one server. Therefore, the algorithm is restricted to running on the english Wikipedia corpus.
It appears that the "skip slope" metaphor with the shapes was a bad idea.
Each shape is a ski-slope difficulty rating symbol. So, Green -- "easy" -- will take the meaning of your query more literally. Double black diamond -- "advanced expert" -- will try to extrapolate hidden meanings in your query. It will suggest topics that are less obviously related to your query.
Re: A new experimental "similarity search" algorithm. Thoughts?
#10It like google with schizophrenia. Pretty impressive. It seems like drugs and alcohol as mind enhancer for creative people (writers, designers, painters) is obsoleted now :-)