Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Are Lucene/Solr/ES Still Used for Search?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 223 posts

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#61

Post by gfs »

deathtrader666 wrote: post #58
lm28469 wrote: post #6

Afaik almost everything runs Lucene under the hood, it's 20 years old, no one is going to build something as good any time soon. I suppose some company like Google have their own in house solution but otherwise it'll always be something built on top of Lucene. I guess you don't see much demand because for a lot of use cases the basic setups are good enough.

Wonder what DuckDuckGo might have built for its search..

I see their logo on the main Solr page [0]. They must be using it in some capacity still.

[0]: https://lucene.apache.org/solr/

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#62

Post by rickmode »

bryanrasmussen wrote: post #52
eivarv wrote: post #30

Earlier quoted context omitted.

Agreed re: using ES as primary storage (which it is NOT meant as) - as far as I can tell, it might even make you in breach of the GDPR [0]. TLDR: Lucene Delivering a conference talk [1] later this year about it. [0]: https://www.eivindarvesen.com/blog/2018/09/16/elasticsearch-... [1]: https://2019.javazone.no/program/3f7cd8a7-a9ea-4874-a7dd-531...

I think the instructor's response in the linked article is a reasonable defense, you don't really know that data is deleted all the way down to the file level. It is just marked as deleted and could be retrieved by someone clever enough to do so. At some point in the future it will be really deleted. I don't think the GDPR regulatory agencies are operating at a technical level that they would make an argument that it…

When using ES for indexing and not the primary store, you can (and should) periodically fully reindex the data set. You can use a blue / green pattern — create a new index then swap from the old one to the new one. ES supports aliases, making this swapping transparent to the apps using the index. Now you have more options.

If it is easy to delete specific users from the primary database, the deleted users will naturally disappear during the next ES reindex.

Edit: The old index is deleted at the file system level.

If the reindexing occurs daily or weekly, perhaps this will satisfy GDPR.

There are other good reason to not use ES as the primary data store. First, it isn’t entirely reliable. It’s good and I’ve never seen a corruption, but ES and Lucene’s history isn’t as a reliable database. Second, if you want to change how you index, it is a bit easier to do if the source data is outside of ES.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#63

Post by rickmode »

balfirevic wrote: post #54
eivarv wrote: post #30

Earlier quoted context omitted.

Agreed re: using ES as primary storage (which it is NOT meant as) - as far as I can tell, it might even make you in breach of the GDPR [0]. TLDR: Lucene Delivering a conference talk [1] later this year about it. [0]: https://www.eivindarvesen.com/blog/2018/09/16/elasticsearch-... [1]: https://2019.javazone.no/program/3f7cd8a7-a9ea-4874-a7dd-531...

> Agreed re: using ES as primary storage (which it is NOT meant as) - as far as I can tell, it might even make you in breach of the GDPR [0]. How is GDPR compliance of having data in Elastic Search influenced by it being primary vs. secondary storage?

See my reply to bryanrasmussen for a full explanation.

Basically: you reindex ES periodically, so when a user is deleted from the primary, it will disappear from ES upon the next reindex. The old index is deleted at the file system level.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#64

Post by bogomipz »

crawdog wrote: post #27

Very much so. For retail/catalog search SOLR dominates. There's a lot more customization available for relevancy/ranking OOB than Elastic. Drawbacks are managing indexing - SOLR cloud is much harder to manage. For commodity search workloads (general retrieval/faceting) Elastic does a fine job. It scales well and there is good documentation and support. Lucene is the core engine behind both of these solutions. For fun…

>"For retail/catalog search SOLR dominates."

Interesting, could you elaborate on why SOLR is dominant in that space over say Elasticsearch?

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#65

Post by busterarm »

devonkim wrote: post #59
showerst wrote: post #50

Earlier quoted context omitted.

I think it depends massively on how much data you have. A great many companies and websites only need a few hundred megs of text data indexed, which is easier to outsource. Once you grow larger than that though, the hosted service prices get astronomical compared to standing up a cluster, assuming you have someone who can admin it.

Things are not quite cut and dry operationally. Some places may have only a few hundred MB of data but they need really high availability and performance (probably picked the wrong solution IMO but still...) which is better guaranteed hosting it yourself and perhaps even outside the cloud. Most places the availability of whatever a hosted solution provides is a better value than spending engineering hours to maintain…

The only issue I have with this suggestion is that all of the situational awareness needed to _use_ Elasticsearch effectively is the same as is needed to _operate_ Elasticsearch effectively.

If you're just spinning up something's defaults and throwing data in it, that bill is going to due eventually and it's probably going to be ugly.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#66

Post by ipython »

rickmode wrote: post #63
balfirevic wrote: post #54

Earlier quoted context omitted.

> Agreed re: using ES as primary storage (which it is NOT meant as) - as far as I can tell, it might even make you in breach of the GDPR [0]. How is GDPR compliance of having data in Elastic Search influenced by it being primary vs. secondary storage?

See my reply to bryanrasmussen for a full explanation. Basically: you reindex ES periodically, so when a user is deleted from the primary, it will disappear from ES upon the next reindex. The old index is deleted at the file system level.

At some point, though, the pedantry can get out of hand. After all, 'deleting' at the file system level is just 'unlinking' the inode from the underlying data blocks... in fact, data forensics at the file system level is probably more well-understood than recovering deleted data from a Lucene shard.

at what point would you be able to 'delete' data without being in violation of GDPR?

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#67

Post by amelius »

busterarm wrote: post #46
jillesvangurp wrote: post #28

Elasticsearch, definitely. I always recommend using it in hosted form and not running your own cluster. That allows you to focus on getting data in and out of your cluster instead of sinking time into doing devops. While I have not used Solr recently, it has evolved along with ES as a solid product with a solid community. Nothing against it; it's a solid choice and there are probably people offering to host that as w…

> Elasticsearch, definitely. I always recommend using it in hosted form and not running your own cluster. That allows you to focus on getting data in and out of your cluster instead of sinking time into doing devops. I think everyone doing Elasticsearch well has to bring it in-house eventually. AWS's hosted solution is poor, Logz.io and ElasticCloud are expensive. There's a 7-figure/yr Elastic Cloud customer I work w…

Also, some clients may prohibit uploading of data to third parties. In those cases you simply have no other choice than to run your own cluster.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#68

Post by crawdog »

bogomipz wrote: post #64
crawdog wrote: post #27

Very much so. For retail/catalog search SOLR dominates. There's a lot more customization available for relevancy/ranking OOB than Elastic. Drawbacks are managing indexing - SOLR cloud is much harder to manage. For commodity search workloads (general retrieval/faceting) Elastic does a fine job. It scales well and there is good documentation and support. Lucene is the core engine behind both of these solutions. For fun…

>"For retail/catalog search SOLR dominates." Interesting, could you elaborate on why SOLR is dominant in that space over say Elasticsearch?

Definitely. Product catalog information as a whole doesn't change often. Price and availability does. With retail catalogs you often do a full reindex of the data in your master catalog and then run partials to account for price/availability if you don't do that in realtime using filters. Since the system of record is not the search index, Elastic is often not a good solution here.

Also, relevancy in retail is often influenced by other factors that cannot easily be implemented in Elastic. TFIDF/BM25 search is available in both platforms, but you may also weigh in other factors such as relationships with the vendor, stock on hand, or other ML techniques that are more easily implemented in SOLR.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#69

Post by rhacker »

parthdesai wrote: post #20
linsomniac wrote: post #9

About a year ago I set up an ES cluster to load Apache logs into as an experiment. Around a month later my boss asked if it was down. "Yeah, it looks like it, since you are using it let me upgrade it from an experiment to critical!" Since then we've been using it in more and more places, and are looking to see if it would be a good fit for storing some of our user data in. The big barrier right now is that we are run…

>The SQL server is always going to be our source of truth, probably. ES is not supposed to be your source of truth.

Definitely agree - ES is designed more like a search appliance - it definitely should be pushed data from other databases that are the source of truth.

https://discuss.elastic.co/t/elasticsearch-as-a-primary-data...

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#70

Post by crawdog »

crawdog wrote: post #68
bogomipz wrote: post #64

Earlier quoted context omitted.

>"For retail/catalog search SOLR dominates." Interesting, could you elaborate on why SOLR is dominant in that space over say Elasticsearch?

Definitely. Product catalog information as a whole doesn't change often. Price and availability does. With retail catalogs you often do a full reindex of the data in your master catalog and then run partials to account for price/availability if you don't do that in realtime using filters. Since the system of record is not the search index, Elastic is often not a good solution here. Also, relevancy in retail is often…

One more point - if you can run the entire index on one machine it makes deploying and managing SOLR much easier to manage than Elastic. The complexity only grows when you have a distributed system. You can fit a lot on a big box.
Post reply on HN