I work professionally in this space and I can say over the past ~10 years, my full time job at 4 companies has been almost entirely to migrate away from SOLR / Lucene solutions and implement custom in-house search indexes. Most of the time it has been for performance reasons. SOLR / Lucene have very poor performance characteristics, especially when needing to support custom sort ordering and heavy use of filters. On…
Ask HN: Are Lucene/Solr/ES Still Used for Search?
201–210 of 223 posts
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#202Earlier quoted context omitted.
* Fast - bought by Microsoft (Also known as the Enron of Norway...) That one was painful to live through, we got forced to migrate to Windows and everything went sideways. That was almost 10 years ago with quite a big cluster (tens of nodes).
You were never forced to migrate to Windows. In fact, the last major customers on ESP were using Linux to the end.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#203We use Solr to power our main, end user-facing search after migrating from a custom Lucene solution some years ago.
To me it seems Lucene based tools are the best for the job if the main thing you care for is having text focused search with a huge potential for extensibility.
But there are a lot of use cases where you will never need anything more than the base capabilities of this technology (so you can be served by something simpler to use or maintain nowadays) and there are probably a lot of use cases where your search will be mainly driven by vector similarity (in which case you are working around the limitations of picking a technology with another focus).
As far as jobs go, I'm not sure how in demand specialists are. After a few years of working in the field I had a look to see if I could leverage my experience to get a remote position and came up with pretty much nothing.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#204Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#205From my experience, yes. Lucene is a "production" state of art library and Solr/Elasticsearch is very used in many scenarios. This expertise is very on demand. My company personally migrated from ElasticSearch to https://vespa.ai/ and could not be happier. Faster and way easier to maintain a cluster. The "Application Packages" feature present in Vespa opened many opportunities to improve our product( Curiously we use…
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#206From my experience, yes. Lucene is a "production" state of art library and Solr/Elasticsearch is very used in many scenarios. This expertise is very on demand. My company personally migrated from ElasticSearch to https://vespa.ai/ and could not be happier. Faster and way easier to maintain a cluster. The "Application Packages" feature present in Vespa opened many opportunities to improve our product( Curiously we use…
It's also readily apparent that it's an ancient system that's grown out of an in-house project, and that its design has accrued a lot of oddities over the years from lack of careful, co-ordinated design. It includes a bunch of esoteric features (like the "predicate" function) that have obviously grown out of Yahoo's own internal architecture.
One curiosity is Vespa's approach to schema and configuration changes. To make any kind of change, or indeed set up an index, you have to create that "application package" containing your schema and configuration in the form of files, and then use separate REST APIs to "upload", "prepare" and "activate" it. There's a CLI tool to help perform those steps, at least.
It's nice that they're more consistent and rigid about schema and config evolution than Elasticsearch. But it's not exactly operator-friendly, at least not for first-time users with no pre-existing operations based around Vespa.
The package design also makes it more cumbersome to perform programmatic updates for a schema. I once worked on a SaaS project where we indexed data in Elasticsearch — arbitrary documents where we didn't know the schema ahead of time, because we just accepted any JSON document posted by the client. With ES, we could just use its dynamic mapping support, which automatically creates field definitions when new fields arrive (using regex-based templates). Do you know how long a package update takes in Vespa, to add, say, a single field?
The Vespa documentation is also pretty terrible, in my opinion. They explain a lot of things, but it's confusingly written, uses a lot of homegrown terminology, and neglects to collect all the reference documentation in one place. For example, you can't find an overview of the entire API — fragments of it are just scattered across a dozen or so unrelated pages.
Lastly, Vespa is Java. One of the biggest challenges maintaining Elasticsearch is controlling its resource consumption. You have to give it a lot of RAM, and it's never clear how much it needs and what configuration settings and usage patterns affect its memory use. Tuning it is something of a dark art. I don't know exactly how Vespa is implemented (is it all pure Java?), but I'm worried that, being a JVM app, it has the same shortcomings.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#207I work professionally in this space and I can say over the past ~10 years, my full time job at 4 companies has been almost entirely to migrate away from SOLR / Lucene solutions and implement custom in-house search indexes. Most of the time it has been for performance reasons. SOLR / Lucene have very poor performance characteristics, especially when needing to support custom sort ordering and heavy use of filters. On…
Are you seeing a blueprint start to emerge for a standardized way to build and deploy search indices in the context of applications that need vector-space features? (E.g. if you start with ANNOY, you get kNN but then how do you add in the ability to refine, filter, rescore, sort with text, etc?)
Instead, you want to custom build the data retrieval system so it’s tailored to your use case.
One example from experience was needing to add hard filterable metadata to an in-house search index. We solved this by actually calculating bit masks that represented all the filtering criteria and having a frontend preprocessor that would first restrict to the filtered subset and then do TFIDF-based relevance sorting.
Creating the bit mask tooling ourselves (instead of relying on whatever baked-in method of scanning items and filtering that comes with out of the box search engine tools) allowed us complete control over the trade-offs, particularly managing document deletions and optimizing run time performance in certain ways that just weren’t available in out of the box tools, as well as being able to integrate any in-house code into the search engine as needed (since the whole system was in-house code).
You want to create data models that are highly application specific, and then route data into them. The mistaken approach of one-size-fits-all tools, especially in information retrieval, is to pre-define the supported behavior of the application, like a web service wrapping a search index, with baked-in assumptions about the trade-offs and only limited support to modify or configure the trade-offs under the hood.
The gravest mistake is thinking just because your use case seems to function OK with those assumptions now, that you can marry yourself to the underlying data model. Then in the future you’ll hit the point where you have to throw it away and create something custom, but it will be far more costly to do so and extremely hard to migrate gracefully and ensure integrations are working.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#208From my experience, yes. Lucene is a "production" state of art library and Solr/Elasticsearch is very used in many scenarios. This expertise is very on demand. My company personally migrated from ElasticSearch to https://vespa.ai/ and could not be happier. Faster and way easier to maintain a cluster. The "Application Packages" feature present in Vespa opened many opportunities to improve our product( Curiously we use…
I've looked at Vespa a bit. It looks pretty good. It's also readily apparent that it's an ancient system that's grown out of an in-house project, and that its design has accrued a lot of oddities over the years from lack of careful, co-ordinated design. It includes a bunch of esoteric features (like the "predicate" function) that have obviously grown out of Yahoo's own internal architecture. One curiosity is Vespa's…
In my own attempt to compare the two, I found the memory consumption of Vespa was easier to predict and understand (there are formulas for it in the documentation).
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#209Two years ago I decided to go with Postgres' built-in fulltext search instead of adding another dependency like ElasticSearch, and I believe I've profited from that in much less maintenance while still getting quite good performance/features.
Do you use ts_rank? PostgreSQL FTS is very efficient until you want to rank the results according to their relevance. This is because the data necessary to the ranking are not in the GIN or GIST index. They are in the heap, and this triggers a lot of random IOs.
EDIT: This seems to help with the ranking problem: https://github.com/postgrespro/rum
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#210Two years ago I decided to go with Postgres' built-in fulltext search instead of adding another dependency like ElasticSearch, and I believe I've profited from that in much less maintenance while still getting quite good performance/features.
Any tips for scaling Postgres-only fulltext search?