Too bad they have yet to address the split brain issue.
True, that's a valid issue. For me, it's not as I end up indexing the same document multiple times over the course of 2-3 days.
Elasticsearch 1.0.0 released
61–70 of 139 posts
Re: Elasticsearch 1.0.0 released
#62We wrote a tutorial about how we wrote our search for Close.io using elasticsearch and pyparsing:
"Sales data search: Writing a query parser / AST using pyparsing + elasticsearch"
Part 1: http://blog.close.io/sales-data-search-writing-a-query-parse...
Part 2: http://blog.close.io/sales-data-search-writing-a-query-parse...
Re: Elasticsearch 1.0.0 released
#63Earlier quoted context omitted.
I can't comment much on MongoDB, but I've written a bit things to keep in mind when considering Elasticsearch as a NoSQL store here: https://www.found.no/foundation/elasticsearch-as-nosql/
An interesting read, but I'd disagree with your contention that NoSQL isn't about ACID. When NoSQL databases started coming out, it was really about which CAP guarantee a database chooses to compromise. Traditional SQL databases are either partition-intolerant or become unavailable (for writes) in the event of a partition. NoSQL databases compromise on consistency. If a database is claiming to be NoSQL and have ACID…
Thanks for the feedback!
The point of that section is exactly that "NoSQL" (or to make things even more confusing "NOSQL" (Not only) doesn't have a very specific meaning. Some think it rules out ACID, other's don't. Thus, you'll need to know what you need.
And database marketing tend to not be very good at pointing out what they're not good at, or actually deliver what they promise. See also: http://aphyr.com/tags/jepsen
Re: Elasticsearch 1.0.0 released
#64Earlier quoted context omitted.
Complexity. Having two copies of the data means more dev time, more resources required to shift the data around, etc. Having just 1 data store that can also handle all your searching is like the holy grail. As you say, not sure if Solr/ES/Lucene are there yet - but they're definitely very very close. There is no theoretical barrier either - it just comes down to closing bugs, and the ES/Lucene team are very good at c…
Sure, that's a fair point. Data consistency reliability in ES and Lucene will only get better over time. But I personally suspect Lucene won't ever get away from the dreaded "just reindex." And to the larger point, I think recent resurgent interest in data stores and distributed systems have shown pretty clearly that there is no holy grail. No single data store can provide all the semantics necessary for all use case…
I'd like to learn from you about "general-purpose primary store". Do you mean for storing any type of data? Here is what I think regarding the case you brought up in the previous post:
ES is suitable for full-text based document indexing for enterprise level or any websites, which means they have a reasonable amount of data to be indexed in a given timeframe. A complete re-indexing won't not take for a couple of days.
So the basic idea behind the NoSQL database is to dump the data into the database quickly and return, so you can see very fast response for insert and delete. Then it will load the data into the memory to process for real-time retrieval which also produces fast response from select. I'm not sure about update.
If the data volume grows, they quickly add shards or make the number of pre-shards big enough to allocate enough memory resources to handle the queries or let the OS to swap the memories by adding more server nodes.
So if you want to use NoSQL database, you must be bound with the system requirement and make your application fit into that and take the most advantage from it. Otherwise, if you are running high structured data store, better to use relational database.
Another point is: if the documents are collected from the web like search engine, NoSQL will not fit for the large volume of data and relational database is also used to store the indexed data for fast retrieval. I guess this is what you meant "general-purpose primary store".
Correct me if I'm wrong.
Re: Elasticsearch 1.0.0 released
#65Elasticsearch is really awesome for searching, but what most people don't realize is that it makes a better MongoDB than MongoDB while giving you that searching too.
This. A THOUSAND TIMES "This". The one drawback ES had in the bad old days was that backup and restore was a nightmare... ESPECIALLY on AWS. The new system they introduced was so simple I was concerned about updating to it because I was SURE something would go south. But it all just worked. I still have the Couch to ES replication running because I'm anal like that... but really... yeah... you can do without Couchbas…
As a proud user of Elastic search since the early days I'm happy to see so much progress. Never mind about the *search part of their naming it's really a database for all practical purposes, especially for web data.
Re: Elasticsearch 1.0.0 released
#66Why is it awesome? Why "it just works"? Is it just a mongodb-kind document store over Hadoop+Lucene?
What makes it so special to have hundreds of votes and tweets all around within 2 hours?
I don't understand. A DB engine engineer.
Re: Elasticsearch 1.0.0 released
#67I also took a few days a few weeks ago to setup elastic search after my mysql full text search fell apart. What I'm doing is slamming the full text output of OCRed PDFs into a MyISAM table, the entire document in a text field. What I'm afraid I'm not doing right is creating the web interface to search elasticsearch. What I'm using filters with the query string syntax[1] in the search box, pointing directly at that fu…
May I ask what you meant about "web based search implementation using elasticsearch"? Do you mean that you use ES to do indexing on the backend of your documents and make it available on the web? Or do you mean that you use ES to index documents available on the web and let people to search for them?
Sure. Your first guess is correct - I do indexing of backend documents.
I fetch a steady stream of FOIA documents, close to the maximum possible each week, and PDF/OCR them. I expose a web interface to the analysts I work with, to help them gather up documents for further analysis.
The second guess would probably be more interesting to most people.
Re: Elasticsearch 1.0.0 released
#68Earlier quoted context omitted.
I'm surprised so many people miss this. Out of the box, Elasticsearch is a distributed NoSQL store with better write consistency (and arguably performance) than MongoDB offers in its default configuration. The major missing feature was backup snapshots and restores, which 1.0 delivers—along with aggregations that more than rival MongoDBs. The team has intentionally avoided marketing themselves as a NoSQL store (was t…
It's easy to miss. On the front page, the word "store" only occurs once, buried three page-scrolls down in the body text. Otherwise it very much gives the impression of being some kind of analytics dashboard for third-party datastores. And I didn't notice that until after I've visited the website, clicked through a few links trying to figure out what the fuss was about, then gave up and decided to read the comments h…
Probably because some store features have been missing up to 1.0, like backup/restore without knowing database internals. (yes, rsync did the job, but only because you knew the list of guarantees that makes it possible).
Also, Lucene at its core is an Index. Changing the query strategy might require reindexing. It is perfectly valid to throw data at it, build the index and throw away the source. You will just never get it back again.
While ES can be used and tuned as a store just fine, it is not necessarily its raison d'etre.
Re: Elasticsearch 1.0.0 released
#69ES seems to have ability to run analytic queries. I have read about people using it as an OLAP solution [1], although I have not yet read anyone describe their experience. In that respect how does ES analytics capabilities compare against:
1) Dremel clones [2] like Impala & Presto (for near real-time, ad hoc analytic queries over large datasets)
2) Lambda Architecture [3] systems (where queries are known up- front, but need to run against a large dataset)
Does anyone here have experience ES in such usecases, beyond the free text searching one ES is well-known for?
[1]: https://groups.google.com/forum/#!topic/elasticsearch/iTy9IY...
[2]: http://static.googleusercontent.com/media/research.google.co...
[3]: http://jameskinley.tumblr.com/post/37398560534/the-lambda-ar...
Re: Elasticsearch 1.0.0 released
#70Elasticsearch mostly "just works". The latest version of Solr has made clustering easier (requires managing Zookeeper), but before that, it was either ES or nightmare. Lucene is one of those projects which hardly has any real competition. That's surprising given how many real world software projects have a search requirement. While Lucene is excellent, it's not without flaws and competition is always great.
I agree with you, almost every website needs a search server on the backend for people to search their document base, especially for enterprise intranet. Maybe enterprises are using commercial products, such as SharePoint. How about the rest of the small businesses and websites? Maybe the learning curve is steep for every website to adopt so far.