Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elasticsearch 1.0.0 released

elasticsearch.org

Post reply on HN
Print view View original

41–50 of 139 posts

Re: Elasticsearch 1.0.0 released

#41

Post by rch »

troels wrote: post #16
m0th87 wrote: post #5

It was two weeks ago, and our startup was on the precipice of a major launch. We had completely rewritten our online publication site, which drives the bulk of our traffic. The product had to be shipped on-time - we had press releases, eager investors and a launch party dependent on it. A few days before launch, things were not looking good. As admins manipulated articles in preparation for the launch, the servers ke…

Did you try/consider Sphinx? It's simple and it's quite fast. I'm using that and I'm pretty happy with it, but I might investigate ES at some point to see if I can squeeze a bit more speed out of it.

You might also take a look at the search functionality in Riak. I've run both Solr and ES, the latter at significant scale, and I'm leaning more towards Riak going forward. The difference is mainly convenience, so not a reason to switch off something that's working already.

Re: Elasticsearch 1.0.0 released

#42

Post by polyfractal »

bryanh wrote: post #27

The thing that worried me the most about Elasticsearch was how fragile it got around the limits of its performance. Run out of memory because of a nasty query? Boom, data corrupted. I hope you weren't using it as your primary persistence layer... Otherwise, we love ES. The other comment about it being a better Mongo than Mongo rings true. With the backup/restore API and the some of the circuit breakers, I'm hopeful t…

FWIW, this is a place ES devs are spending a lot of time thinking about. For example, 1.0 introduces a new "Circuit Breaker" [1] feature which will help prevent over-eager facets from blowing out the heap. It's just one part of a very large effort to make ES handle exceptional events more gracefully (in particular, memory related).

Another example are disk-based doc values [2], which are essentially pre-computed field data structures that are stored on disk. This moves Field Data off heap and allows the OS to manage memory evictions, to help minimize GCs and OOM blowouts.

[1] http://www.elasticsearch.org/guide/en/elasticsearch/referenc...

[2] http://www.elasticsearch.org/blog/disk-based-field-data-a-k-...

Re: Elasticsearch 1.0.0 released

#43

Post by Argorak »

Beyond the technology, Elasticsearch has a very mature, active and helpful community with users groups all over the world. We're well connected.

Pick your favourite users group here: http://elasticsearch.meetup.com/

Full disclosure: I started and run the Berlin UG. We set ourselves apart by always providing a small introduction into ES for those that are completely new and would have a hard time following the main talk.

Re: Elasticsearch 1.0.0 released

#44

Post by alisson »

axefrog wrote: post #18
RyanZAG wrote: post #7

Elasticsearch is really awesome for searching, but what most people don't realize is that it makes a better MongoDB than MongoDB while giving you that searching too.

What limitations should one be aware of that would make ElasticSearch not a viable candidate where something like MongoDB would be a better fit?

On ElasticSearch you have to update the whole document, no commands to manipulate them. You don't have commands like: $set, $addToSet, $pop, etc..

You need to have a good understanding of how tokenizers and analyzers work to be able to create good results for your data. I have difficulties matching documents with the exact title being searched for. On MongoDB that just works, on ElasticSearch you need to configure it.

ElasticSearch has some advantages and MongoDB others. I think they are great together. One for storage and the other for searching.

Re: Elasticsearch 1.0.0 released

#45

Post by polyfractal »

alisson wrote: post #44
axefrog wrote: post #18

Earlier quoted context omitted.

What limitations should one be aware of that would make ElasticSearch not a viable candidate where something like MongoDB would be a better fit?

On ElasticSearch you have to update the whole document, no commands to manipulate them. You don't have commands like: $set, $addToSet, $pop, etc.. You need to have a good understanding of how tokenizers and analyzers work to be able to create good results for your data. I have difficulties matching documents with the exact title being searched for. On MongoDB that just works, on ElasticSearch you need to configure it…

Regarding updates, you can use the Update API for partial updates, and include a script to do things like "counter += 1" or "add value to existing array".

Internally it is still reindexing the entire document, but from your application's perspective, the Update API is a lot friendlier.

http://www.elasticsearch.org/guide/en/elasticsearch/referenc...

Re: Elasticsearch 1.0.0 released

#46

Post by NDizzle »

nzadrozny wrote: post #30
NDizzle wrote: post #19

I also took a few days a few weeks ago to setup elastic search after my mysql full text search fell apart. What I'm doing is slamming the full text output of OCRed PDFs into a MyISAM table, the entire document in a text field. What I'm afraid I'm not doing right is creating the web interface to search elasticsearch. What I'm using filters with the query string syntax[1] in the search box, pointing directly at that fu…

I host and support websolr.com and bonsai.io and have seen a lot of search implementations. The main thing for good stability and performance is to be very good at batching your updates. You don't want to sling a ton of highly-parallel single-document updates at Lucene, lest you thrash the JVM and start garbage collecting like crazy. From there, on the query side, you'll want to get a good working knowledge of the di…

Do you have a demo on either of those sites where I can input terms into a search box and look at results? What explanation do you give to users as to the options available when formatting the query?

Re: Elasticsearch 1.0.0 released

#47

Post by acdha »

dc2447 wrote: post #8
m0th87 wrote: post #5

It was two weeks ago, and our startup was on the precipice of a major launch. We had completely rewritten our online publication site, which drives the bulk of our traffic. The product had to be shipped on-time - we had press releases, eager investors and a launch party dependent on it. A few days before launch, things were not looking good. As admins manipulated articles in preparation for the launch, the servers ke…

> Elasticsearch mostly "just works", and we didn't have to worry about complex schema definitions, working with giant complex XML files (hello Solr) If you were using Solr there are a few operational modes to run in. Config file based or SolrCloud[0]. The latter is more akin the ES in terms of cluster management. I agree though from an simplicity of deployment perspective at scale ES is has a much lighter learning cu…

SolrCloud is nothing like ES in terms of management: you end up running a separate zookeeper service with even more files which all have to be configured correctly just to get it running and you have to micromanage shard allocation to ensure that you can add nodes in the future but also not have it intentionally deadlock when a server fails and you no longer have enough nodes for a quorum. All of this happens with the usual contempt for sysadmins where things you need to know (“refusing to process requests”) won't be logged but a bunch of startup boilerplate will be, and simply configuring logging correctly requires (IIRC) editing two XML files and a properties file.

`java -jar elasticsearch.jar` does a better job and that's basically all it takes. I'm planning to switch as soon as https://github.com/elasticsearch/elasticsearch/issues/256 lands.

Re: Elasticsearch 1.0.0 released

#48

Post by nzadrozny »

RyanZAG wrote: post #36
nzadrozny wrote: post #29

Earlier quoted context omitted.

Ditto open file handles, which is easy to push when aggressively over-sharding. Not an uncommon mistake for the enthusiastic newbie. Having supported Solr/ES/Lucene in production for 4+ years now (websolr.com / bonsai.io) I would be pretty hesitant to trust Lucene in general as a primary data store. Beautiful for secondary indexing, but otherwise, Why Not Postgres?™ ;)

Complexity. Having two copies of the data means more dev time, more resources required to shift the data around, etc. Having just 1 data store that can also handle all your searching is like the holy grail. As you say, not sure if Solr/ES/Lucene are there yet - but they're definitely very very close. There is no theoretical barrier either - it just comes down to closing bugs, and the ES/Lucene team are very good at c…

Sure, that's a fair point. Data consistency reliability in ES and Lucene will only get better over time.

But I personally suspect Lucene won't ever get away from the dreaded "just reindex." And to the larger point, I think recent resurgent interest in data stores and distributed systems have shown pretty clearly that there is no holy grail. No single data store can provide all the semantics necessary for all use cases. Maybe not even for most use cases. There are just too many tradeoffs to consider.

Believe me, I earn a living hosting Elasticsearch, so I'd love to see it become a robust primary data store. There are some use cases where it actually does make sense—just look at the amazing traction ES is experiencing for storing and indexing time-series data.

But as a general-purpose primary store, I'm not really holding my breath. Maybe I'm just becoming battle-worn and bitter. I would love to be proven otherwise over the next few years!

Re: Elasticsearch 1.0.0 released

#49

Post by nzadrozny »

NDizzle wrote: post #46
nzadrozny wrote: post #30

Earlier quoted context omitted.

I host and support websolr.com and bonsai.io and have seen a lot of search implementations. The main thing for good stability and performance is to be very good at batching your updates. You don't want to sling a ton of highly-parallel single-document updates at Lucene, lest you thrash the JVM and start garbage collecting like crazy. From there, on the query side, you'll want to get a good working knowledge of the di…

Do you have a demo on either of those sites where I can input terms into a search box and look at results? What explanation do you give to users as to the options available when formatting the query?

We've got a free Heroku addon that's pretty easy to spin up and play with. Elasticsearch also has an analyze[1] API that can be helpful to play around with.

It's also possible to download and install ES locally and run any number of front-end interfaces, some of which include query builders. ElasticHQ seems like a decent option for that. The venerable Elasticsearch-head is another.

I think now that ES 1.0 has shipped, more experimental tools will start to emerge that help people learn and interact with ES itself. (If anyone out there is a front-end whiz and wants to help me build something like that, please email nz@bonsai.io!)

1. http://www.elasticsearch.org/guide/en/elasticsearch/referenc...

Re: Elasticsearch 1.0.0 released

#50

Post by brasetvik »

RyanZAG wrote: post #36
nzadrozny wrote: post #29

Earlier quoted context omitted.

Ditto open file handles, which is easy to push when aggressively over-sharding. Not an uncommon mistake for the enthusiastic newbie. Having supported Solr/ES/Lucene in production for 4+ years now (websolr.com / bonsai.io) I would be pretty hesitant to trust Lucene in general as a primary data store. Beautiful for secondary indexing, but otherwise, Why Not Postgres?™ ;)

Complexity. Having two copies of the data means more dev time, more resources required to shift the data around, etc. Having just 1 data store that can also handle all your searching is like the holy grail. As you say, not sure if Solr/ES/Lucene are there yet - but they're definitely very very close. There is no theoretical barrier either - it just comes down to closing bugs, and the ES/Lucene team are very good at c…

Sometimes, I actually find it easier to have more systems that do their job really well and sync things between them, rather than trying to get a single system to do everything.

For example, Postgres lets you reason about integrity, atomicity and transactional boundaries, and whether things are really safely stored with synchronous replication. If Postgres returns after a commit, I trust it. However, that requires me to have two servers working, which is harder to keep highly available.

ZooKeeper, on the other hand, I can rely on being available. But that's not really something you want to be putting lots of load on, nor try to do anything but trivial "queries". And the more servers you add, the slower writes get.

I don't trust Elasticsearch enough for those tasks, yet I wouldn't want to do searches in Postgres (Yep, I'm familiar with tsearch) even though it can. Elasticsearch is simple to scale out and awesome for searching.

Logs and metrics we shove straight into Elasticsearch, however. Other things go from ZooKeeper to Postgres and then to Elasticsearch, or from just Postgres to Elasticsearch.

Separate tools for separate jobs. I'm one of the co-founders of www.found.no, one of the hosted Elasticsearch providers . We absolutely love Elasticsearch and find new use cases for it all the time, but it's not going to be the one store to rule them all, at least not very soon.

Post reply on HN