Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Solr 8.6.1

mail-archives.apache.org

Post reply on HN
Print view View original

11–20 of 24 posts

Re: Solr 8.6.1

#11

Post by tekkk »

Solr is one of those technologies which works but isn't really glorious to use and is bit stuffy with its XML configurations and Java interfaces. It's a bit of shame, because search engines are so popular nowadays and everybody seems to be fixated on using ElasticSearch. Which from what I've read and heard is resource-hungry and not really cut for simple text-search.

Maybe someday somebody will create a new search engine that will be hipster and easy to use, like Algolia but open-source. I'm bit curious though how incumbent Lucene is as the core search engine that it would even make sense to try and recreate it in another language. Probably too much that unless somebody has money to throw around it will remain so to the far future.

Re: Solr 8.6.1

#12

Post by hermanradtke »

tekkk wrote: post #11

Solr is one of those technologies which works but isn't really glorious to use and is bit stuffy with its XML configurations and Java interfaces. It's a bit of shame, because search engines are so popular nowadays and everybody seems to be fixated on using ElasticSearch. Which from what I've read and heard is resource-hungry and not really cut for simple text-search. Maybe someday somebody will create a new search en…

And the need for ZooKeeper with SolrCloud. We had to have at least one person working full time to keep a decent sized cluster working correctly.

Re: Solr 8.6.1

#13

Post by cmckn »

joking wrote: post #9
leetrout wrote: post #8

Earlier quoted context omitted.

Could you speak more about using solr as a primary DB? Or did I misunderstand how you were using it?

you can, but almost nobody will recommend to use it, neither will do with elasticsearch. In most cases it works, but it has priorized speed over resilience, so it's better to have a source from where you can rebuild the index.

> it's better to have a source from where you can rebuild the index.

Having worked for several years at a company selling search, I can't emphasize this enough. Rebuilding indices should always be straightforward, and your data should be very accessible in some other (preliminary) form. We ran into so many issues, and had so many panics, because the index had become the only place some things existed. It's also way easier to tweak (read: optimize) your schema over time when you're in the habit of rebuilding the index.

Re: Solr 8.6.1

#14

Post by csunbird »

hermanradtke wrote: post #12
tekkk wrote: post #11

Solr is one of those technologies which works but isn't really glorious to use and is bit stuffy with its XML configurations and Java interfaces. It's a bit of shame, because search engines are so popular nowadays and everybody seems to be fixated on using ElasticSearch. Which from what I've read and heard is resource-hungry and not really cut for simple text-search. Maybe someday somebody will create a new search en…

And the need for ZooKeeper with SolrCloud. We had to have at least one person working full time to keep a decent sized cluster working correctly.

This mirrors my experience. The Solr Cluster that we have in production requires full time attention of our infrastructure team.

Re: Solr 8.6.1

#15

Post by haddr »

csunbird wrote: post #14
hermanradtke wrote: post #12

Earlier quoted context omitted.

And the need for ZooKeeper with SolrCloud. We had to have at least one person working full time to keep a decent sized cluster working correctly.

This mirrors my experience. The Solr Cluster that we have in production requires full time attention of our infrastructure team.

I had an opposite experience: solr was rock solid and didn’t require too much maintenance.

Re: Solr 8.6.1

#16

Post by eivarv »

joking wrote: post #9
leetrout wrote: post #8

Earlier quoted context omitted.

Could you speak more about using solr as a primary DB? Or did I misunderstand how you were using it?

you can, but almost nobody will recommend to use it, neither will do with elasticsearch. In most cases it works, but it has priorized speed over resilience, so it's better to have a source from where you can rebuild the index.

Another "but" here is what [0] may happen if you're storing personal data in big indexes...

TLDR; In some cases, what data you actually have (as in "on disk") becomes complicated. For instance, in some cases, once a segment reaches max size (5GB by default) it will only be eligible for merging when it accumulates 50% deletions. This means you might not be able to guarantee deletion of data within 30 days (in accordance with the GDPR).

[0]: https://www.eivindarvesen.com/blog/2018/09/23/lucene-indexes...

Re: Solr 8.6.1

#17

Post by softwaredoug »

Solr is great, but it's frustratingly unopinionated about a lot of things.

Should you use SolrCloud or Solr standalone mode? Use the classic config files? Or dynamic/managed schemas and config APIs?

You may say "to each their own" but when you want to do a simple thing, like install a plugin, and there's about 5 ways to do it - with each way working for a different 75% of use cases, it can be rather frustrating. Compare this to Elasticsearch where there's one company behind everything (for better or worse) with very strict and clear opinions about how to do something.

I'm working on a little project with my personal opinions for Solr use. Call it Effective Solr, or Solr The Good Parts. Who knows when/if it'll get done, and they're my somewhat informed opinions, but maybe someone would find them useful.

It's rather early, but it never hurts to begin gathering comments on this

https://gist.github.com/softwaredoug/3212fa9c5a198a565a9a77b...

Re: Solr 8.6.1

#19

Post by koffiedrinker »

dimitar wrote: post #18

Does Solr have what the HFT guy calls 'catastrophic typing' in ES? https://thehftguy.com/2020/08/04/the-differences-between-spl...

Yes, since you have to define the type of each field. If you have a managed schema (=you don't define a schema but let SolR auto-create it), SolR will pick the type for you. Any document with a field not matching the type will be rejected.

Re: Solr 8.6.1

#20

Post by lukevp »

tekkk wrote: post #11

Solr is one of those technologies which works but isn't really glorious to use and is bit stuffy with its XML configurations and Java interfaces. It's a bit of shame, because search engines are so popular nowadays and everybody seems to be fixated on using ElasticSearch. Which from what I've read and heard is resource-hungry and not really cut for simple text-search. Maybe someday somebody will create a new search en…

Search really has 3 main levels.

1. Core ranked inverted index data store (Lucene)

2. API layer on top of Lucene with easier query, index, data importing, sharding, etc. (Solr/Elastic Search)

3. Fully hosted API / UI for easier GUI for developers and search relevance engineers (Algolia, Lucidworks Fusion).

The new hotness in search is currently Rust-based tooling. Rust is a great application for search as it's very performance-sensitive and data-structure heavy, and once the indexes are built are fairly stable, leading to the 90th+ percentile latencies and throughput to be much better than Lucene-based libraries that are built on top of the JVM.

For the low level core search engine like Lucene (inverted indexes, TF-IDF ranking, etc), there's Tantivy [1]

For the middle tier (ES) there's Sonic [2]

MeiliSearch [3] is a play for the hipster open-source Algolia. It's in Rust, is MIT licensed, supports self-hosting, has an out of the box web interface.

[1] https://github.com/tantivy-search/tantivy [2] https://github.com/valeriansaliou/sonic [3] https://github.com/meilisearch/MeiliSearch

Post reply on HN