Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Full Text Search in Mongo

mongodb.org

Post reply on HN
Print view View original

11–15 of 15 posts

Re: Full Text Search in Mongo

#11

Post by rabidsnail »

ChrisFulstow wrote: post #7
rabidsnail wrote: post #6

Earlier quoted context omitted.

How are you doing your tokenization and stemming? I find it hard to believe that the actual token lookup is slower.

Tokenization was a simple string split on whitespace, and no stemming. It was quite a large Mongo dataset, so only a fraction of the index and data would've been in memory, it could've easily been quicker for a smaller dataset living in memory. For me, one of the benefits of Lucene is the powerful built-in query parsing, tokenization, analyzers, etc.

The performance probably would have been better if the dataset (or at least the portion of the dataset that gets touched frequently) was smaller. But why wouldn't lookup in lucene be at least as slow?

Re: Full Text Search in Mongo

#12

Post by vain »

i was very excited by this technology. i tried to implement it on a large database. it bombed, the hype around nosql kind of whitewashes the fact that at the end, it is indexed in a btree, exactly how mysql would do it. i am not trashing mongodb, i am a big fan of it, i am just making a point about being objective.

Re: Full Text Search in Mongo

#13

Post by rb2k_ »

kunjaan wrote: post #10
DanielRibeiro wrote: post #3

Earlier quoted context omitted.

More like throwing some wood into the fire. Was instigated by the talk[1] "Solr Power FTW", and "Building a recommendation engine, foursquare style" [6] (where Justin Moore admits: We are dumping the data from Mongo and loading it into Hadoop over S3 files. Map reduce is in this system, not in our mongo databases. ). Was wondering how much has this evolved over the last 6 months, how viable machine learning algorithm…

The question regarding the difference between CouchDB and Mongo wasn't fully explored. Can someone here comment one which route makes more sense right now? Riak also seems to have full-text search. Has anyone used it?

Riak Full text search is fun to work with and as simple to set up as the rest of Riak. They offer a Solr interface, but don't support all of the operations yet (e.g. http://wiki.basho.com/Riak-Search---Querying.html#Faceted-Qu... --> Facet querying through the Solr interface is not yet supported. ).

Riak isn't the fastest single node system, but if you're going big and need several servers anyway it will save you some time.

CouchDB could use elasticsearch and its streaming indexation ("river" -> http://www.elasticsearch.org/docs/elasticsearch/river/couchd... ) to get scalable fulltext search.

An interesting project for fulltext search when it comes to mongodb and SOLR is photovoltaic ( https://github.com/mikejs/photovoltaic ), it pipes the mongoDB changes to the Solr XML interface. Sadly, I haven't had time to use it yet, but it looks interesting.

Re: Full Text Search in Mongo

#14

Post by rb2k_ »

ChrisFulstow wrote: post #5

This isn't really supposed to be a proper full-text index feature, instead it's building a rudimentary inverted index using a string array property. It's possible to create indexes over array properties in MongoDB, which is very cool, and increases performance to an extent. But even with an index, this approach to full-text was much slower for me than an equivalent search against the same data in Lucene. I'd love to…

> I'd love to see a MongoDB component that replicates data from the oplog to a dedicated full-text store like Lucene or Solr.

Photovoltaic does this: https://github.com/mikejs/photovoltaic

I haven't had time to play arround with it yet though :(

Re: Full Text Search in Mongo

#15

Post by FooBarWidget »

vain wrote: post #12

i was very excited by this technology. i tried to implement it on a large database. it bombed, the hype around nosql kind of whitewashes the fact that at the end, it is indexed in a btree, exactly how mysql would do it. i am not trashing mongodb, i am a big fan of it, i am just making a point about being objective.

You're right, that is what many NoSQL databases are. However this simplicity also allows NoSQL databases to more easily implement features like sharding, which is why they're useful. I've yet to see an SQL database that supports sharding and doesn't cost a lot of money.
Post reply on HN