Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudant (YC S08) Releases In-Database, Distributed Search

blog.cloudant.com

Post reply on HN
Print view View original

11–20 of 21 posts

Re: Cloudant (YC S08) Releases In-Database, Distributed Search

#11

Post by dcaylor »

timanglade wrote: post #9
dcaylor wrote: post #7

How does this compare to the various existing Lucene based search options for CouchDB? The post says the new Cloudant Search "is a way that would not require you to set up a third-party, financially or operationally expensive solution." Adding basic Lucene searches to a CouchDB setup isn't all that hard. What about elasticsearch and solr? Aside from the cost and hosting, are there other differentiaters between Clouda…

I already responded to this partly in http://news.ycombinator.com/item?id=2764736 — We think the integration into a single deployment is in itself a big gain. Maintaining several of those infrastructures can be very painful, especially as your clusters grow large. Also, as mentioned, we’ve also already added several features on top of Lucene, and we’ll be adding more in the future.

Yes, thank you. Just after I posted my question here I realized that much of what I was wondering about was also answered in another post on the Cloudant blog: http://blog.cloudant.com/technical-look-at-cloudant-search/

Re: Cloudant (YC S08) Releases In-Database, Distributed Search

#12

Post by paulasmuth »

timanglade wrote: post #8
paulasmuth wrote: post #6

Wouldn't it be possible to do the same thing with Apache Lucene/Solr?

Not entirely. The big difference here is that you only have one, integrated architecture to maintain. So that means less operational complexity, smoother “scalability” of your infrastructure, tighter integration between DB & Search. Also, we added stuff like queries across several indices, typed queries, etc.

Hm, as far as I understand multiple-index (core) search has been implemented in Solr 1.3?

Re: Cloudant (YC S08) Releases In-Database, Distributed Search

#13

Post by hardtke »

paulasmuth wrote: post #12
timanglade wrote: post #8

Earlier quoted context omitted.

Not entirely. The big difference here is that you only have one, integrated architecture to maintain. So that means less operational complexity, smoother “scalability” of your infrastructure, tighter integration between DB & Search. Also, we added stuff like queries across several indices, typed queries, etc.

Hm, as far as I understand multiple-index (core) search has been implemented in Solr 1.3?

Sharded search is not new. Solr, elastic-search, and Riak do it as well. The difference here is that we've built the Search on top of the BigCouch map-reduce view model. Views are calculated post commit so there are no data insertion locks. Multiple copies of each shard exist for fault-tolerance. Also, multiple map-reduce analytics passes can be used as input to the search.

Re: Cloudant (YC S08) Releases In-Database, Distributed Search

#14

Post by mark_l_watson »

owenmarshall wrote: post #3

Will this make its way upstream into the open-source BigCouch?

While I share your sentiment that it would be nice to have as part of BigCouch, that is probably asking a lot since this is a good value add for their (Cloudant's) service.

BTW, I have only set up BigCouch a couple of times to play with it but it is very impressive.

Re: Cloudant (YC S08) Releases In-Database, Distributed Search

#16

Post by gniquil »

I have a very noob question. For most databases, suffix search is always super slow. However, can't someone just build an index based on the string reversed, then treat suffix search exactly the same way as prefix search? This doubles your index storage requirement. But index storage is generally not a problem. Finally this could be perhaps extended to cover any wildcard searches (hell*world)

Re: Cloudant (YC S08) Releases In-Database, Distributed Search

#17

Post by davisp »

brendoncrawford wrote: post #15

Maybe slightly off-topic here, but are there plans to eventually merge Big Couch upstream into Couch core?

Its hard to say. There are quite a few ways in which this could play out.

Firstly, there are two important points to consider. Currently, BigCouch is more or less a superset of Apache CouchDB. The only patches we have to CouchDB sources can and should be back-ported but require that we solve a couple possibilities for bugs in non-clustered deployments. Secondly, Erlang is a language which allows for an easy mish-mashing of code so that once we have back-ported these patches there's no real requirement for a merge at all.

There are also a few things that we're discussing in the CouchDB community that could very well contribute to not needing to merge the projects. Specifically rearranging our source tree to be more prototypically Erlang as well as some tools like a couch-config script that could allow plugin-type extensions to CouchDB.

In the end, its hard to tell how things will shape up. It could be a full on back-port, or it could just be a general improvement to CouchDB's source tree and build system so that BigCouch is strictly "CouchDB + Other Erlang Apps" if that makes sense. And with my CouchDB committer hat on, it really depends on what the community wants. Its easy to fall into think of the trap of "it's obvious" but we also have to consider that others are taking CouchDB and porting it to mobile phones. What we end up with in "core" CouchDB has to consider a lot of use cases.

Re: Cloudant (YC S08) Releases In-Database, Distributed Search

#18

Post by mronge »

gniquil wrote: post #16

I have a very noob question. For most databases, suffix search is always super slow. However, can't someone just build an index based on the string reversed, then treat suffix search exactly the same way as prefix search? This doubles your index storage requirement. But index storage is generally not a problem. Finally this could be perhaps extended to cover any wildcard searches (hell*world)

Yep. That is exactly how you solve this problem. I've done this with Lucene and then searched either the regular field or the reversed field depending on the wildcard query.

Re: Cloudant (YC S08) Releases In-Database, Distributed Search

#19

Post by davisp »

gniquil wrote: post #16

I have a very noob question. For most databases, suffix search is always super slow. However, can't someone just build an index based on the string reversed, then treat suffix search exactly the same way as prefix search? This doubles your index storage requirement. But index storage is generally not a problem. Finally this could be perhaps extended to cover any wildcard searches (hell*world)

The issue there is that you're still anchoring your index to one end of the string which means you're not solving the general problem, only a specific manifestation of it.

A general example would be given the string "foo bar baz", your solution could find "foo%" or "%baz" efficiently, but not "%bar%". Its not out of the question if what you really want is a suffix search, but the general problem of finding an internal substring is still less than optimal.

Edit: Formatting

Post reply on HN