Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elasticsearch 1.0.0 released

elasticsearch.org

Post reply on HN
Print view View original

71–80 of 139 posts

Re: Elasticsearch 1.0.0 released

#71

Post by dclara »

NDizzle wrote: post #67
dclara wrote: post #59

Earlier quoted context omitted.

May I ask what you meant about "web based search implementation using elasticsearch"? Do you mean that you use ES to do indexing on the backend of your documents and make it available on the web? Or do you mean that you use ES to index documents available on the web and let people to search for them?

Sure. Your first guess is correct - I do indexing of backend documents. I fetch a steady stream of FOIA documents, close to the maximum possible each week, and PDF/OCR them. I expose a web interface to the analysts I work with, to help them gather up documents for further analysis. The second guess would probably be more interesting to most people.

Yes, then I think ES fits our application well and you should really take its advantage to provide your web interface for searching those documents.

I'm more interested in the second case, but I don't think ES fits due to the huge volume of data to be indexed.

Re: Elasticsearch 1.0.0 released

#72

Post by troels »

rch wrote: post #41
troels wrote: post #16

Earlier quoted context omitted.

Did you try/consider Sphinx? It's simple and it's quite fast. I'm using that and I'm pretty happy with it, but I might investigate ES at some point to see if I can squeeze a bit more speed out of it.

You might also take a look at the search functionality in Riak. I've run both Solr and ES, the latter at significant scale, and I'm leaning more towards Riak going forward. The difference is mainly convenience, so not a reason to switch off something that's working already.

Hadn't considered Riak, but I can see that it has some full-text search capabilities. Any idea about its features and how it compares in performance, as a raw search index?

Re: Elasticsearch 1.0.0 released

#74

Post by sandstrom »

This gem is from the 'breaking changes' list:

  “Geo queries used to use miles as the default unit. And we 
  all know what happened at NASA because of that decision. The
  new default unit is meters.”
I like this release already.

Re: Elasticsearch 1.0.0 released

#75

Post by ddorian43 »

AznHisoka wrote: post #58
ddorian43 wrote: post #20

Earlier quoted context omitted.

presharding You create a number of shards for each index(database) that you can't later expand.

What I've done, and I'm not totally sure if it's a best practice is I've over-allocated the # of shards. So if I think I need 5 shards, I create 50 or 100 shards instead. Then I'll have some app logic to determine the shard a document should go to. Initially all docs will go to shard 0. Then when that's full (around 15 GB of size, depends on your RAM), then I set all docs to go to shard 1. Of course, you'll need to b…

look: routing_field

Re: Elasticsearch 1.0.0 released

#76

Post by ddorian43 »

RyanZAG wrote: post #34
ddorian43 wrote: post #20

Earlier quoted context omitted.

presharding You create a number of shards for each index(database) that you can't later expand.

Is this still a limitation? I haven't run into any use cases where this has been a problem yet. Since the default shards are 10 and 2 replicas, does that not mean each index should be able to scale up to 20 servers? I'd think that if your data grew enough that 1/10th does not fit on a server, you could do a one time maintenance and rebuild all your servers. I have my doubts mongodb would scale up that well to 20+ ser…

you can grow a little larger than that by using some nodes only for aggregating/handling queries(holding no data/shards)

larger number of shards=slower searching (unless you distribute the shards to multiple nodes)

Re: Elasticsearch 1.0.0 released

#77

Post by NDizzle »

dclara wrote: post #71
NDizzle wrote: post #67

Earlier quoted context omitted.

Sure. Your first guess is correct - I do indexing of backend documents. I fetch a steady stream of FOIA documents, close to the maximum possible each week, and PDF/OCR them. I expose a web interface to the analysts I work with, to help them gather up documents for further analysis. The second guess would probably be more interesting to most people.

Yes, then I think ES fits our application well and you should really take its advantage to provide your web interface for searching those documents. I'm more interested in the second case, but I don't think ES fits due to the huge volume of data to be indexed.

Oh - I have one! I just want to see examples of others so I can figure out ways to improve my implementation.

Re: Elasticsearch 1.0.0 released

#78

Post by ddorian43 »

axefrog wrote: post #18
RyanZAG wrote: post #7

Elasticsearch is really awesome for searching, but what most people don't realize is that it makes a better MongoDB than MongoDB while giving you that searching too.

What limitations should one be aware of that would make ElasticSearch not a viable candidate where something like MongoDB would be a better fit?

also changing indexed-fields on the go

Re: Elasticsearch 1.0.0 released

#79

Post by capkutay »

I was vetting ES for a business critical search platform, had some concerns about write/read performance and how the lucene indexes are handled on disk. I read that it doesn't really perform as well a splunk...Instead of ES, I'm considering a solution using HBase to shard lucene indexes on HDFS.

Re: Elasticsearch 1.0.0 released

#80

Post by gibrown »

alecco wrote: post #66

Why is it awesome? Why "it just works"? Is it just a mongodb-kind document store over Hadoop+Lucene? What makes it so special to have hundreds of votes and tweets all around within 2 hours? I don't understand. A DB engine engineer.

There are a lot of features thoughtfully combined that make ES great. Top of my list would be:

1. It handles human written language. Any language. The same technology that let's it handle strings written in human language provides a lot of flexibility in handling string in other applications. Particular when handling logs.

2. Non-string data it also handles very fast and cleanly (numbers, dates, geo).

3. Lucene has an inverted index that has been optimized over many years. ES scales that pretty seamlessly across many servers. All decisions in the project seem to be made around whether a feature can scale to 100s of nodes.

The devs have also been really smart to focus on the "out of box experience". Very well thought out defaults.

More on our experience with ES at scale: http://gibrown.wordpress.com/2014/01/09/scaling-elasticsearc...

Post reply on HN